PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Sketch-Image Hashing

Yuming Shen, Li Liu, Fumin Shen, Ling Shao

arXiv:1803.02284v1cs.CV

TL;DR

The paper addresses zero-shot SBIR hashing, where test categories are unseen during training and existing methods generally fail. It proposes ZSIH, a three-network generative hashing model that combines modality-specific encoders, fusion, graph convolution, and semantic reconstruction, and reports significant improvements over existing methods.

  • Problem

    Zero-shot SBIR hashing is needed because existing methods generally fail when test categories are excluded from training.

  • Method

    ZSIH uses an end-to-end three-network architecture with image and sketch encoders, a training-time multi-modal network, and semantic-aware generative hashing.

  • Results

    ZSIH significantly outperforms existing methods on the zero-shot SBIR hashing task.

  • Takeaways & Limitations

    The model provides a unified approach combining cross-modal hashing, SBIR, and zero-shot learning.

  • Takeaways & Limitations

    The method assumes seen-category training data, semantic representations, and completely disjoint unseen test categories.

Abstract

from arXiv · show

Recent studies show that large-scale sketch-based image retrieval (SBIR) can be efficiently tackled by cross-modal binary representation learning methods, where Hamming distance matching significantly speeds up the process of similarity search. Providing training and test data subjected to a fixed set of pre-defined categories, the cutting-edge SBIR and cross-modal hashing works obtain acceptable retrieval performance. However, most of the existing methods fail when the categories of query sketches have never been seen during training. In this paper, the above problem is briefed as a novel but realistic zero-shot SBIR hashing task. We elaborate the challenges of this special task and accordingly propose a zero-shot sketch-image hashing (ZSIH) model. An end-to-end three-network architecture is built, two of which are treated as the binary encoders. The third network mitigates the sketch-image heterogeneity and enhances the semantic relations among data by utilizing the Kronecker fusion layer and graph convolution, respectively. As an important part of ZSIH, we formulate a generative hashing scheme in reconstructing semantic knowledge representations for zero-shot retrieval. To the best of our knowledge, ZSIH is the first zero-shot hashing work suitable for SBIR and cross-modal search. Comprehensive experiments are conducted on two extended datasets, i.e., Sketchy and TU-Berlin with a novel zero-shot train-test split. The proposed model remarkably outperforms related works.

1. Introduction

The paper formulates zero-shot SBIR hashing for retrieval when test categories are unseen during training, addressing both category exclusion and efficient cross-modal matching. It proposes ZSIH, an end-to-end generative hashing model that combines binary encoding, heterogeneity mitigation, and semantic transfer.

  • Motivation: Existing SBIR and cross-modal hashing methods generally fail when retrieval categories were absent from training because the learned model lacks conceptual knowledge about what to find.
  • Task formulation: Conventional SBIR and cross-modal hashing assume training categories include the test categories, whereas zero-shot SBIR uses disjoint seen and unseen categories.In the illustrated setting, training uses classes A and B while test sketches and images come from categories such as plane and cat.
  • Contributions: The model formulates deep generative hashing to transfer semantic knowledge to unseen classes while producing efficient binary representations.
  • Contributions: ZSIH is presented as the first zero-shot hashing work for large-scale SBIR.
  • Contributions: ZSIH uses an end-to-end three-network structure with attention, Kronecker fusion, and graph convolution to handle category exclusion and sketch-image heterogeneity.
  • Results: ZSIH achieves reasonable retrieval performance under the zero-shot setting, while existing methods generally fail.

2. The Proposed ZSIH Model

ZSIH learns image and sketch hashing functions from seen-category multimodal data and semantic representations, then encodes unseen-category data with disjoint test classes. Its design targets zero-shot retrieval using shared binary codes.

  • Problem setting: The model trains on paired image-sketch data from seen categories and requires semantic representations to transfer supervised knowledge to unseen data.
  • Problem setting: ZSIH learns separate deep hashing functions for images and sketches that encode unseen test data into M-bit binary codes.
  • Problem setting: The zero-shot protocol requires no intersection between seen training categories and unseen test categories.
  • Architecture: ZSIH is an end-to-end architecture composed of image and sketch encoders plus a multi-modal network.

2.1. Network overview

The network overview combines modality-specific binary encoders with a training-only multi-modal network. Attention, Kronecker fusion, graph convolution, and generative decoding jointly support modality alignment and semantic-aware hashing.

  • Image/sketch encoding networks: Image and sketch encoders use attention models to pool informative feature regions before sigmoid-based binary encoding.The attention mechanism produces a 256-D feature, which feeds a fully connected binary-encoding layer.
  • Multi-modal network: During training, a multi-modal network learns joint sketch-image representations and addresses modality heterogeneity through Kronecker-product fusion.The fusion layer combines outputs from the image and sketch attention models for each paired data point.
  • Semantic modeling: Graph convolution incorporates semantic relations so knowledge from seen categories can generalize to unseen categories.
  • Generative hashing: A stochastic generative hashing module produces binary-code supervision by decoding and reconstructing semantic information.The decoder is optimized through decoding likelihood with gradient-based methods.

2.2. Fusing sketch and image with Kronecker layer

The Kronecker fusion layer combines attention-derived sketch and image features into a high-dimensional representation designed to preserve expressive cross-modal structure.

  • Attention outputs h(sk) and h(im), each in R256, are fused for a same-category sketch-image pair.
  • The fusion uses a third-order parameter tensor and tensor dot products before Tucker decomposition produces the fused representation.
  • The Kronecker operation applies trainable 256×256 transformations to the two modalities, followed by ReLU activation.The resulting feature vector has 65536 dimensions.
  • Compared with concatenation or factorized fusion, the Kronecker layer expands hidden-state dimensionality with relatively few parameters and stores more expressive sketch-image relations.

2.3. Semantic-relation-enhanced hidden representation with graph convolution

ZSIH uses graph convolution on in-batch multimodal representations, with adjacency determined by semantic representations, to intensify relations among data points.

  • Each training batch is treated as an NB-vertex graph whose hidden states are processed by graph convolutional layers.The layer maps H^l to H^(l+1) through a parameterized convolutional filter.
  • The graph convolutional propagation uses a self-connected in-batch adjacency matrix A and its degree matrix D.The adjacency is determined by semantic representations of the batch samples.
  • Two graph convolutional layers output dimensions NB×1024 and NB×M, using ReLU first and sigmoid second to restrict final values to 0–1.
  • Graph convolution performs row transformations according to the graph Laplacian, intensifying semantic relations in hidden states for zero-shot hashing.

2.4. Stochastic neurons and decoding network

ZSIH converts multimodal representations into stochastic binary codes and decodes them through a semantic-regularized generative hashing network trained end-to-end.

  • The second graph-convolution output produces latent probabilities b∈(0,1)^M for a sketch-image pair and its semantic feature.
  • Stochastic neurons sample binary codes eb∈{0,1}^M from the latent probabilities using random thresholds.
  • The training procedure samples mini-batches of same-class sketch-image pairs, constructs semantic-distance adjacency, samples codes, and updates parameters until convergence or the maximum iteration.
  • The posterior of b given an image and sketch is approximated by a Multinoulli distribution to support differentiable back-propagation through the decoder.
  • A decoder is placed above the stochastic neurons and regularized by semantic representations through a Gaussian likelihood with reparameterization.
  • The decoder parameters μ(·) and σ(·) are fully connected layers, enabling end-to-end network training.

2.5. Learning objective and optimization

ZSIH jointly learns semantic-aware binary encoders using an objective that combines generative semantic reconstruction with sketch-image code alignment, optimized end-to-end.

  • The learning objective is defined over batches of sketch and image data to train the whole network jointly.
  • The expectation term maximizes code entropy and preserves semantic knowledge through the decoder’s semantic likelihood.
  • L-2 losses align image and sketch encoder outputs with the sampled code, reflecting sketch-image similarity in shared codes.
  • Semantic-aware training codes allow the encoders to represent out-of-sample data without additional category information.
  • ZSIH estimates gradients by Monte Carlo sampling and trains efficiently end-to-end with SGD rather than alternating optimization.

2.6. Out-of-sample extension

ZSIH generates binary codes for unseen-class sketches and images using separate encoding networks, without requiring test-data semantic representations or multimodal processing.

  • ZSIH hashes unseen-class image and sketch data for matching after training.
  • Separate encoding networks f(·) and g(·) produce the test codes independently.Test semantic representations are unnecessary, and test encoding can therefore be efficient.

3. Experiments

Experiments evaluate ZSIH for zero-shot cross-modal retrieval on extended Sketchy and TU-Berlin datasets using Hamming-space search and comparisons with hashing, SBIR, and zero-shot baselines. ZSIH generally outperforms the selected methods, while ablations highlight the contributions of its fusion, semantic, and graph components.

  • Datasets and baselines: Experiments use extended Sketchy and TU-Berlin datasets with unseen-category splits and nearest-neighbour retrieval in Hamming space.Sketchy uses 25 unseen test classes and 100 seen training classes; TU-Berlin uses 30 test classes with the remainder for training.
  • Datasets and baselines: The evaluation compares ZSIH with cross-modal hashing, SBIR, and zero-shot learning baselines retrained or configured under the same zero-shot setting.Baselines include DSH, ZSH, CMT, DeViSE, SAE, Siamese CNN, GN Triplet, and related methods.
  • Comparison with existing methods: ZSIH significantly outperforms selected hashing baselines, while DSH’s conventional-SBIR advantage drops dramatically on unseen categories.ZSH performs relatively better than other hashing baselines but was designed for single-modal retrieval.
  • Comparison with existing methods: ZSIH maintains a similar performance margin over existing SBIR and zero-shot learning methods, many of which use inefficient real-valued representations.High-dimensional continuous-space nearest-neighbour search reduces retrieval efficiency.
  • Ablation study: 64-bit mAP results show that feature concatenation, MFB, and bit-regularization baselines underperform the original ZSIH design.Bit regularization also produces unstable training and can lead to overfitting.
  • Ablation study: Figure 5 examines 32-bit retrieval across different numbers of seen classes and visualizes seen and unseen classes with t-SNE.ZSIH achieves acceptable retrieval when an adequate number of seen classes is used during training.
  • Ablation study: Removing graph convolution reduces mAP by about 4%, while replacing the semantic decoder with a classifier causes a dramatic performance fall.The authors associate graph convolution with enhanced semantic relations and the decoder with generalization from seen to unseen classes.

4. Conclusion

The paper introduces zero-shot SBIR hashing and addresses it with ZSIH, an end-to-end architecture for shared binary representations. Its Kronecker fusion, graph convolutions, and generative hashing scheme support heterogeneous sketch-image retrieval, and experiments report significant gains over existing methods.

  • ZSIH tackles efficient large-scale zero-shot SBIR hashing with an end-to-end three-network architecture for shared binary representations.The architecture encodes sketch and image data through image/sketch encoders and a multimodal network.
  • The model mitigates sketch-image heterogeneity with attended-feature Kronecker fusion and introduces semantic knowledge through graph convolutions and generative hashing.
  • Experiments report that ZSIH significantly outperforms existing methods on the zero-shot SBIR hashing task.
Loading 1803.02284v1…