PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrieval

Anjan Dutta, Zeynep Akata

arXiv:1903.03372v1cs.CV

TL;DR

Sketch-based image retrieval matches free-hand sketches to relevant natural images, providing a useful alternative when textual descriptions or suitable query images are unavailable. SEM-PCYC learns unaligned sketch and image mappings into a shared semantic space, and the paper reports consistent outperformance of existing methods on two datasets.

  • Problem

    Sketch-based image retrieval addresses cases where users can sketch a desired object more easily than provide a textual description or suitable image.

  • Method

    SEM-PCYC uses adversarially trained sketch and image mappings, branch-wise cycle consistency, classification losses, and selected side information to form a shared semantic space without aligned sketch-image pairs.

  • Results

    SEM-PCYC consistently outperforms existing methods on two zero-shot SBIR datasets.

  • Takeaways & Limitations

    Cycle consistency and classification losses complement adversarial training, while compact selected side information supports effective zero-shot SBIR.

  • Takeaways & Limitations

    Generalizing unseen classes from representations learned on seen classes remains challenging on TU-Berlin.

Abstract

from arXiv · show

Zero-shot sketch-based image retrieval (SBIR) is an emerging task in computer vision, allowing to retrieve natural images relevant to sketch queries that might not been seen in the training phase. Existing works either require aligned sketch-image pairs or inefficient memory fusion layer for mapping the visual information to a semantic space. In this work, we propose a semantically aligned paired cycle-consistent generative (SEM-PCYC) model for zero-shot SBIR, where each branch maps the visual information to a common semantic space via an adversarial training. Each of these branches maintains a cycle consistency that only requires supervision at category levels, and avoids the need of highly-priced aligned sketch-image pairs. A classification criteria on the generators' outputs ensures the visual to semantic space mapping to be discriminating. Furthermore, we propose to combine textual and hierarchical side information via a feature selection auto-encoder that selects discriminating side information within a same end-to-end model. Our results demonstrate a significant boost in zero-shot SBIR performance over the state-of-the-art on the challenging Sketchy and TU-Berlin datasets.

1. Introduction

Zero-shot SBIR targets retrieval when test classes are unseen, making domain gaps, intra-class variability, and limited unseen-class knowledge central challenges. SEM-PCYC addresses these issues by learning semantically aligned sketch and image mappings without requiring aligned pairs.

  • Motivation: Zero-shot SBIR seeks accurate class prediction and retrieval when training and test classes do not overlap.The setting is more realistic than conventional SBIR, which benefits from overlapping training and test classes.
  • Motivation: Zero-shot SBIR is challenging because it combines a significant domain gap, intra-class variability, and limited knowledge about unseen classes.
  • Limitations of prior work: Paired sketch-image supervision can improve multimodal learning but is often unavailable or expensive to obtain.
  • Limitations of prior work: High-dimensional memory fusion can cause information loss when useful multimodal information is extracted.
  • Proposed approach: SEM-PCYC maps sketch and image features to a common semantic space using adversarial training and cycle consistency, avoiding aligned sketch-image pairs.Classification loss makes generated semantic features discriminative, while a feature-selection autoencoder combines textual and hierarchical side information.
  • Evaluation: The model is evaluated on Sketchy and TU-Berlin, where it consistently improves state-of-the-art zero-shot SBIR results.

2. Related Work

Prior work addresses sketch-image domain differences through handcrafted features, cross-domain deep learning, and zero-shot semantic mappings. Zero-shot SBIR combines these retrieval and transfer-learning challenges while using side information to transfer knowledge to unseen classes.

  • Sketch-Based Image Retrieval: Handcrafted SBIR methods extract edge-based features and match sketches with natural images using models such as Bag-of-Words.
  • Sketch-Based Image Retrieval: Neural SBIR methods learn domain-transferable features end to end, with many using siamese networks and cross-modal retrieval losses.
  • Zero-Shot Learning: Zero-shot learning recognizes objects from unseen classes by mapping visual features to semantic or shared intermediate spaces.
  • Zero-Shot Learning: Side information transfers supervised knowledge from seen classes to unseen classes in zero-shot learning.
  • Zero-Shot Sketch-Based Image Retrieval: Zero-shot SBIR combines zero-shot learning with sketch-based image retrieval, including generative and autoencoder-based approaches for semantic alignment.

3. SEM-PCYC Model

SEM-PCYC learns sketch and image mappings into a shared semantic space using adversarial training, cycle consistency, and classification supervision, without requiring aligned sketch-image pairs. It also selects discriminative textual and hierarchical side information through an end-to-end autoencoder.

  • Paired Cycle Consistent Generative Model: Cycle consistency translates each modality to the semantic space and back, requiring only category-level supervision rather than aligned sketch-image pairs.The sketch and image training data need not be aligned, provided same-category samples share category labels.
  • Paired Cycle Consistent Generative Model: SEM-PCYC learns separate sketch-to-semantic and image-to-semantic mappings that share a common semantic space.The model also uses reverse mappings to reconstruct the original modalities.
  • Paired Cycle Consistent Generative Model: Adversarial training aligns generated semantic representations with side information while discriminators distinguish original and generated representations.The shared semantic discriminator compares original side information with sketch- and image-generated side information.
  • Paired Cycle Consistent Generative Model: Classification losses on both generators' outputs encourage class-discriminative semantic features for zero-shot retrieval.Adversarial and cycle-consistency objectives alone do not explicitly ensure class discrimination.
  • Selection of Side Information: A feature-selection autoencoder combines textual and hierarchical side information and selects a compressed representation for semantic alignment.The selected representation reduces embedding dimensionality and can improve retrieval time.
  • Training Objective: The full training objective combines adversarial, cycle-consistency, classification, and autoencoder losses in one end-to-end model.The objective includes adversarial losses for semantic and modality mappings, two cycle losses, two classification losses, and the autoencoder loss.

4. Experiments

Experiments on Sketchy and TU-Berlin evaluate SEM-PCYC against adapted SBIR, ZSL, and zero-shot SBIR baselines, then examine side information and loss components. SEM-PCYC improves state-of-the-art mAP@all and shows that complementary semantic embeddings and the full loss combination matter, while TU-Berlin remains challenging because unseen classes can be difficult to generalize from seen classes.

  • Experimental Setup: Experiments use Sketchy and TU-Berlin with seen/unseen class splits, VGG-16 features, attention-based feature extraction, and comparisons against fourteen adapted SBIR and ZSL models.Sketchy uses 25 unseen and 100 seen classes; TU-Berlin uses 30 test and 220 training classes.
  • Comparing with the State-of-the-Art: SEM-PCYC also significantly outperforms the compared method in generalized zero-shot SBIR, where the retrieval gallery contains both seen and unseen classes.This setting is described as more challenging because seen classes distract from test queries.
  • Qualitative Results: TU-Berlin remains difficult because the model must generalize unseen classes from seen-class representations, while qualitative retrieval can confuse visually or semantically similar categories.Examples include tank sketches retrieving motorcycles, guitars retrieving violins, and castles retrieving sky-heavy scenes.
  • Effect of Side-Information: Word2Vec plus Jiang-Conrath reaches 0.349 mAP on Sketchy, while Word2Vec plus path similarity reaches 0.297 mAP on TU-Berlin.The results support complementarity between text-based and hierarchy-based class embeddings, with Word2Vec outperforming GloVe for this task.
  • Model Ablations: Adversarial, cycle-consistency, and classification losses are jointly needed for effective zero-shot SBIR, while removing more than roughly 20%–40% of side information deteriorates performance.Small removals can improve performance by eliminating low-information or noisy components, but further compression removes useful information.

5. Conclusion

SEM-PCYC maps sketch and image features to a common semantic space through adversarial training, while cycle consistency removes the need for aligned sketch-image pairs. It also uses heterogeneous side information and achieves consistent outperformance on two zero-shot SBIR datasets.

  • SEM-PCYC maps sketch and image features to a common semantic space through adversarial training with a shared discriminator.
  • Cycle consistency on both branches removes the requirement for aligned sketch-image pairs and regularizes adversarial training.
  • Classification losses on the generators promote discriminative features for zero-shot SBIR.
  • An autoencoder combines heterogeneous side information into a compact representation useful for adversarial training.
  • The model consistently outperforms existing methods on two zero-shot SBIR datasets.
Loading 1903.03372v1…