PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval

Qing Liu, Lingxi Xie, Huiyu Wang, Alan Yuille

arXiv:1904.03208v3cs.CV

TL;DR

Zero-shot sketch-based image retrieval asks whether models trained on some object categories can retrieve photos for unseen classes, despite domain shift and possible loss of pretrained knowledge. SAKE preserves ImageNet knowledge during fine-tuning through teacher-student supervision refined with semantic information. It achieves state-of-the-art results on TU-Berlin and Sketchy, with benchmark gains also reported for binary codes.

  • Problem

    ZS-SBIR evaluates retrieval on classes unseen during training, where fine-tuned models may overfit the source domain and lose ImageNet knowledge needed to represent target samples.

    Source: §1 · p. 2, §3.2 · p. 3

  • Method

    SAKE uses a fixed ImageNet-pretrained teacher and semantic information to refine teacher-student supervision, preserving knowledge without access to original ImageNet data.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    SAKE achieves state-of-the-art performance on TU-Berlin and Sketchy; on TU-Berlin Extension, it yields 20.9% relative improvement of mAP@all using 64-bit binary hashing codes.

    Source: §1 · p. 2, §4.2 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    Experiments support knowledge preservation as a useful approach for ZS-SBIR, and SAKE requires no extra resources during testing.

    Source: §1 · p. 2, §5 · p. 8

    The authors identify better alignment of sketch and photo features as a possible way to further improve SAKE.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Sketch-based image retrieval (SBIR) is widely recognized as an important vision problem which implies a wide range of real-world applications. Recently, research interests arise in solving this problem under the more realistic and challenging setting of zero-shot learning. In this paper, we investigate this problem from the viewpoint of domain adaptation which we show is critical in improving feature embedding in the zero-shot scenario. Based on a framework which starts with a pre-trained model on ImageNet and fine-tunes it on the training set of SBIR benchmark, we advocate the importance of preserving previously acquired knowledge, e.g., the rich discriminative features learned from ImageNet, to improve the model's transfer ability. For this purpose, we design an approach named Semantic-Aware Knowledge prEservation (SAKE), which fine-tunes the pre-trained model in an economical way and leverages semantic information, e.g., inter-class relationship, to achieve the goal of knowledge preservation. Zero-shot experiments on two extended SBIR datasets, TU-Berlin and Sketchy, verify the superior performance of our approach. Extensive diagnostic experiments validate that knowledge preserved benefits SBIR in zero-shot settings, as a large fraction of the performance gain is from the more properly structured feature embedding for photo images. Code is available at: https://github.com/qliu24/SAKE.

1. Introduction

Zero-shot sketch-based image retrieval (ZS-SBIR) targets retrieval when test classes were unseen during training, where models must bridge sketch-photo modalities and adapt beyond the source domain. The paper attributes weak transfer partly to catastrophic forgetting and introduces SAKE to preserve ImageNet knowledge while fine-tuning; experiments report gains on two SBIR datasets.

1. Introduction

  • SBIR retrieves relevant gallery photos from a hand-drawn sketch query by mapping both modalities into a shared feature embedding.Relevance is based on similar visual content or the same object category.Source: §1 · p. 1
  • ZS-SBIR tests retrieval on object classes unseen during training, unlike conventional SBIR, where training and testing categories overlap.The task requires object recognition, cross-modal matching, and domain adaptation.Source: §1 · p. 1
  • Fine-tuning can erase ImageNet discriminative features, leaving representations less able to distinguish unseen target-domain samples.After fine-tuning, ImageNet classification accuracy falls from 56.29% to 45.54%.Source: §1 · p. 2
  • SAKE preserves pretrained knowledge without access to ImageNet training data, using a fixed pretrained model as a teacher and semantic information to refine its signal.The teacher-student process approximately maps fine-tuning samples into the ImageNet semantic space.Source: §1 · p. 2
  • SAKE boosts ZS-SBIR over state-of-the-art methods on TU-Berlin and Sketchy, with gains persisting after image features are binarized.Training adds moderate computation and little memory, with no extra resources required at test time.Source: §1 · p. 2

2. Related Work

Related work covers SBIR and its zero-shot setting, catastrophic forgetting in fine-tuning, and knowledge distillation. SAKE draws on these areas to preserve pretrained knowledge while targeting generalization to unknown domains.

2. Related Work

  • ZS-SBIR research extends SBIR with semantic embeddings and generative models, including GANs and VAEs, to transfer cross-modal representations to target domains [4] [35] [16].Deep networks followed earlier hand-crafted feature and Bag-Of-Words approaches.Source: §2 · p. 2
  • Catastrophic forgetting describes a fine-tuned model losing its original-domain task ability; prior approaches protect old tasks by slowing selected weights or mimicking model responses [11].SAKE instead targets generalization to unknown domains and refines the original response with semantic constraints [17] [36].Source: §2 · p. 2
  • Knowledge distillation transfers a teacher's knowledge to a student, and SAKE uses pseudo ImageNet labels for fine-tuning samples to preserve pretrained knowledge [12] [31] [40] [1].Source: §2 · p. 2

3. The Proposed Approach

SAKE frames zero-shot sketch retrieval as transfer from a labeled source reference set to a target set with unseen classes, and addresses this through knowledge preservation during fine-tuning. Its approach combines a shared photo-sketch feature network with benchmark classification and semantically guided ImageNet teacher supervision.

3.1. Problem Statement

  • Zero-shot retrieval trains on a source reference set and tests on a target set whose classes do not appear during training.The reference set supplies category labels and cross-modal similarity guidance; testing uses sketch queries to retrieve same-label photos.Source: §3.1 · p. 3

3.2. Motivation: the Connection between Zero-shot Learning and Catastrophic Forgetting

  • Fine-tuning an ImageNet-pretrained AlexNet on TU-Berlin reduced ImageNet classification accuracy from 56.29% to 45.54%, demonstrating that some pretrained knowledge was forgotten.The paper conjectures that alleviating forgetting may strengthen adaptation to the unseen target domain.Source: §3.2 · p. 4

3.3. Semantic-Aware Knowledge Preservation

  • SAKE uses a joint CSE-ResNet-50 with domain-conditioned channel reweighting to process photos and sketches in a shared network.A binary domain code indicates whether the input is a photo or sketch, allowing the network to model modality-specific characteristics.Source: §3.3 · p. 4
  • The feature network feeds benchmark-label and ImageNet-label classifiers, respectively adapting to SBIR data and preserving ImageNet recognition capability.The ImageNet classifier is trained without original ImageNet data, using teacher supervision from the fixed pretrained model.Source: §3.3 · p. 4

3.4. Objective of Optimization

  • SAKE jointly trains a ground-truth benchmark classification loss and a knowledge-preservation loss based on the ImageNet classifier.The benchmark loss uses cross-entropy with benchmark labels, while the knowledge-preservation loss combines teacher predictions with semantic constraints.Source: §3.4 · pp. 4–5
  • WordNet similarities between benchmark and ImageNet classes refine the teacher signal, encouraging ImageNet predictions that align with semantically similar benchmark labels.The semantic-aware logits weight the teacher logits and the benchmark-label semantic similarities; the teacher-only loss is a special case.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate SAKE on TU-Berlin and Sketchy, comparing retrieval performance and testing knowledge preservation through quantitative and qualitative analyses. SAKE outperforms prior ZS-SBIR methods, while diagnostics examine its transfer behavior and learned representations.

4.1. Datasets and Settings

  • Experiments evaluate SAKE on TU-Berlin and Sketchy, randomly selecting 30 and 25 testing classes respectively, with remaining 220 and 100 classes for training; a separate careful Sketchy split uses 21 non-ImageNet classes.TU-Berlin contains 20,000 sketches across 250 categories, and Sketchy contains 75,471 sketches across 125 categories.Source: §4.1 · p. 5

4.2. Comparison with Existing Methods

  • 20.9% relative improvement in mAP@all on TU-Berlin Extension with 64-bit binary codes; SAKE also consistently outperforms state-of-the-art ZS-SBIR methods across datasets and splits.Three random-split runs on TU-Berlin Extension yielded mAP@all values of 0.352, 0.369, and 0.359.Source: §4.2 · p. 6, Table 1 · p. 6
  • On a careful split that tests only non-ImageNet classes, SAKE still outperforms baselines, supporting transfer beyond the ImageNet categories used to pre-train it.The authors report this evaluation for both TU-Berlin and Sketchy.Source: §4.2 · p. 6

4.3. Quantitative Analysis

  • Fine-tuning reduces ImageNet top-1 accuracy, while SAKE recovers some of that loss: AlexNet scores 56.29% pre-trained, 45.54% fine-tuned, and 51.39% with SAKE.For SE-ResNet-50, the corresponding accuracies are 77.43%, 59.56%, and 67.44%; ablations also report a mild semantic-constraint boost when λ1 = 1, peaking at λ2 = 0.3.Source: §4.3 · p. 7
  • The CSE module supports more flexible modality-specific processing, and teacher-based knowledge preservation improves retrieval across backbones, with semantic awareness adding a further boost.The full SAKE model achieves the best retrieval results in the reported backbone and loss-term comparisons.Source: §4.3 · p. 7
  • SAKE’s ZS-SBIR gains are attributed mainly to more structured photo representations; increasing λSAKE improves ImageNet-photo retrieval, while non-ImageNet-photo gains saturate earlier.For ZS-SBIR, performance peaks at λSAKE = 1 and then drops when the teacher signal becomes too influential.Source: §4.3 · pp. 7–8

4.4. Qualitative Analysis

  • Top-10 retrievals are usually category-correct, while selected failures return photos from visually similar but incorrect categories.Retrieval uses cosine distance on 64-d feature vectors.Source: §4.4 · p. 8, Figure 3 · p. 8
  • The t-SNE visualization shows more clearly clustered object classes with SAKE than with the baseline, alongside margins between photo and sketch data.The authors suggest that better alignment of sketches and photos could further improve SAKE.Source: §4.4 · p. 8, Figure 4 · p. 8

5. Conclusions

The paper reframes zero-shot sketch-based image retrieval as incremental learning, using knowledge preservation to alleviate catastrophic forgetting. Experiments on TU-Berlin and Sketchy demonstrate state-of-the-art performance.

5. Conclusions

  • SAKE preserves pretrained-domain knowledge using semantics and works without access to the original training images.Experiments on TU-Berlin and Sketchy demonstrate state-of-the-art performance.Source: §5 · p. 8
  • The paper links zero-shot learning and incremental learning through their shared focus on transferring a trained model to another domain.It conjectures and empirically verifies that improving incremental learning benefits zero-shot retrieval.Source: §5 · p. 8
  • The authors argue that different learning tasks may reflect the same underlying issue, often over-fitting, and propose using one task to assist another.They call for further research in this direction.Source: §5 · p. 8

Appendix A.

The appendix presents additional evidence about how knowledge preservation relates to domain adaptation, focusing on category similarity and SAKE’s performance improvements.

Appendix A.

  • The appendix examines whether target-category similarity to ImageNet categories correlates with SAKE’s zero-shot performance gains.It frames the analysis as further evidence for knowledge preservation benefiting domain adaptation.Source: Appendix A · p. 11

A.1. Setting

The appendix tests category-similarity effects using three ImageNet-disjoint held-out sets and groups categories by SAKE’s retrieval improvement.

A.1. Setting

  • The analysis uses three randomly selected TU-Berlin held-out sets of 30 categories, each with no overlap with ImageNet.The disjoint sets are intended to support fair comparisons across target categories.Source: §A.1 · p. 11
  • Categories are ranked by SAKE’s mAP@all improvement and divided into low-, medium-, and high-improvement groups of 10.The comparison starts from a vanilla SE-ResNet-50 pretrained on ImageNet and fine-tuned on the TU-Berlin reference set with only Lbenchmark.Source: §A.1 · p. 11
  • Figure 5 summarizes the relationship between category-level ImageNet similarity and ZS-SBIR mAP@all improvement, with means and error bars across three held-out sets.All three held-out category sets have no overlap with ImageNet.Source: Figure 5 · p. 11

A.2. Similarity by Classification Confidence

SAKE’s gains are larger for categories with higher ImageNet classification confidence, while improvement is also observed for categories outside ImageNet.

A.2. Similarity by Classification Confidence

  • SAKE’s improvement becomes more significant as a category receives higher classification confidence from an ImageNet-based classifier.The authors interpret this pattern as greater benefit for categories closer to ImageNet, which are often more affected by catastrophic forgetting.Source: §A.2 · p. 11
  • Categories outside ImageNet also gain accuracy under zero-shot evaluation, and the main experiments show benefits for both ImageNet and non-ImageNet categories.Source: §A.2 · p. 11

A.3. Similarity by Semantic-space Distance

Using Leacock–Chodorow similarity over WordNet synsets, the experiment again finds SAKE more effective for categories better represented in ImageNet’s semantic space.

A.3. Similarity by Semantic-space Distance

  • With Leacock–Chodorow similarity over WordNet synsets, SAKE is more effective on categories better represented in ImageNet’s semantic space.The passage reports this as the same conclusion as the earlier experiment.Source: §A.3 · p. 11
Loading 1904.03208v3…