PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Hashing via Transferring Supervised Knowledge

Yang Yang, Weilun Chen, Yadan Luo, Fumin Shen, Jie Shao, Heng Tao Shen

arXiv:1606.05032v1cs.CV

TL;DR

Existing supervised hashing lacks timely, reliable labels for rapidly emerging concepts and therefore needs a way to encode unseen categories from limited seen-category data. ZSH transfers supervision through semantic embeddings, rotates that space to reduce semantic shift, and preserves local structure and discreteness in binary codes; experiments report superior zero-shot image retrieval against several state-of-the-art hashing methods.

  • Problem

    Rapidly emerging concepts and expensive manual labeling make sufficient supervised information unavailable for learning hash functions for new categories.

  • Method

    ZSH projects 0/1 labels into a semantic embedding space, rotates the embedding to align with visual features, and preserves local structure and discrete binary codes.

  • Results

    ZSH shows superior zero-shot image retrieval performance compared with several state-of-the-art hashing methods.

  • Takeaways & Limitations

    Seen-category supervised knowledge can be transferred to hash images from unseen categories without visual observations of those unknown classes.

  • Takeaways & Limitations

    The authors identify incomplete representations from individual modalities as a remaining issue and propose combining textual and visual knowledge in future work.

Abstract

from arXiv · show

Hashing has shown its efficiency and effectiveness in facilitating large-scale multimedia applications. Supervised knowledge e.g. semantic labels or pair-wise relationship) associated to data is capable of significantly improving the quality of hash codes and hash functions. However, confronted with the rapid growth of newly-emerging concepts and multimedia data on the Web, existing supervised hashing approaches may easily suffer from the scarcity and validity of supervised information due to the expensive cost of manual labelling. In this paper, we propose a novel hashing scheme, termed \emph{zero-shot hashing} (ZSH), which compresses images of "unseen" categories to binary codes with hash functions learned from limited training data of "seen" categories. Specifically, we project independent data labels i.e. 0/1-form label vectors) into semantic embedding space, where semantic relationships among all the labels can be precisely characterized and thus seen supervised knowledge can be transferred to unseen classes. Moreover, in order to cope with the semantic shift problem, we rotate the embedded space to more suitably align the embedded semantics with the low-level visual feature space, thereby alleviating the influence of semantic gap. In the meantime, to exert positive effects on learning high-quality hash functions, we further propose to preserve local structural property and discrete nature in binary codes. Besides, we develop an efficient alternating algorithm to solve the ZSH model. Extensive experiments conducted on various real-life datasets show the superior zero-shot image retrieval performance of ZSH as compared to several state-of-the-art hashing methods.

1. INTRODUCTION

Existing supervised hashing enables efficient multimedia retrieval but struggles to generalize when new concepts lack timely manual labels. Zero-shot hashing transfers supervision from seen to unseen categories through semantic embeddings, alignment, and structure-preserving binary-code learning.

  • Motivation: Hashing compresses high-dimensional multimedia data into compact binary codes for efficient storage and Hamming-distance retrieval.It is used for large-scale image and video applications.
  • Problem: Rapidly emerging concepts and the cost of manual labeling leave supervised hashing without sufficient timely training data for new categories.Existing methods may perform well on seen categories but fail to generalize to unseen ones.
  • Proposed direction: Zero-shot hashing learns hash functions from limited seen-category data to encode images from unseen categories.The approach transfers supervised knowledge between seen and unseen classes rather than requiring visual observations of every unseen class.
  • Supervision transfer: ZSH maps independent labels into a semantic embedding space so relationships between seen and unseen categories can support supervision transfer.Semantically related categories can share information through the learned mapping.
  • Hash-function learning: ZSH rotates the embedded space to reduce semantic shift and preserves local structure and discreteness when learning binary codes.The rotation better aligns embedded semantics with low-level visual features, while structural preservation keeps similar data close in Hamming space.

2. RELATED WORK

Related work spans zero-shot learning and hashing methods, but conventional hashing approaches generally remain limited to categories represented during training. This motivates a hashing method that can generalize to unseen categories without frequent retraining.

  • Zero-shot learning: Zero-shot learning uses intermediate visual attributes or semantic descriptors to represent novel images and classify unseen objects.Prior work includes attribute-based descriptions of unseen objects and other semantic representations.
  • Hashing: Hashing research includes unsupervised, multimodal, and supervised methods that use data distributions or auxiliary information to preserve similarity.Examples include Spectral Hashing, Inductive Manifold Hashing, Iterative Quantization, and Sparse Multi-Modal Hashing.
  • Remaining gap: Existing hashing methods fail to generalize to unseen categories and may require expensive retraining as databases change.This limitation motivates data-dependent hashing that works beyond the seen training area.

3. ZERO-SHOT HASHING

The paper presents zero-shot hashing as a framework for learning hash functions that transfer supervision from seen categories to unseen ones. Its pipeline combines semantic knowledge transfer, semantic alignment, hashing, and an optimization procedure.

  • Framework: ZSH is organized around a formal zero-shot hashing definition, framework overview, supervision transfer, semantic alignment, hashing model, and optimization analysis.The method is presented as a complete framework rather than only a retrieval objective.

3.1 Problem Definition

The problem definition separates training and testing concepts: training images belong to seen categories, while test images belong to disjoint unseen categories. The goal is to learn a binary hash function from seen data that preserves semantic similarity and generalizes to unseen images.

  • Problem setting: Training images X are labeled from a seen concept set C, while test images X^(u) belong to a disjoint unseen set C^(u).The condition is C^(u) ∩ C = ∅.
  • Learning objective: Using only seen-category training images, the method learns f: R^d 7→ {−1, 1}^l to map images into l-bit binary codes.No training samples from unseen categories are available during learning.
  • Desired properties: The learned hash function should assign short Hamming distances to semantically relevant objects while generalizing to unseen-category test data.The objective combines semantic retrieval behavior with zero-shot generalization.

3.2 Overall Framework

The framework has offline and online stages: it learns from images in limited seen categories, then supports hashing for new categories. The offline stage extracts visual features and embeds seen labels into a semantic-rich space.

  • Overall Framework: The framework consists of offline and online phases for operating with images from limited visible categories.The supplied passage introduces the two-stage organization and the limited-category setting.
  • Offline Phase: During offline processing, the system extracts visual features from seen images using a convolutional neural network.
  • Offline Phase: It simultaneously transforms seen labels into a semantic-rich embedding space, representing each label as a real-valued vector.

3.3 Transferring Supervised Knowledge

The method replaces independent 0/1-form labels with semantic embeddings that encode relationships among categories. This lets unseen labels use mappings learned from semantically close seen categories.

  • Motivation: Independent 0/1-form labels omit inherent correlations among categories, limiting supervised hashing when queries come from unseen classes.
  • Semantic Embedding: ZSH maps independent labels into a word embedding space where semantic correlations among labels can be quantitatively measured and captured.
  • Knowledge Transfer: Unseen labels can leverage mappings learned from semantically close seen categories, such as a cat benefiting from supervised knowledge of dog images.
  • Embedding Construction: Each category is represented by a 50-d word vector from a language model pretrained on Wikipedia text.

3.4 Semantic Alignment

The method addresses semantic shift by aligning the off-the-shelf embedding space with the distributional properties of low-level visual features. Its formulation maps binary codes to supervisory information through a learned matrix.

  • Semantic Shift: Off-the-shelf embeddings may deviate from image semantics because of domain difference, semantic shift, and semasiological variation.
  • Alignment Strategy: ZSH introduces semantic alignment to align the initial embedding space with the distributional properties of low-level visual features.
  • Model Role: The formulation uses W^T to map binary codes B to supervised information, narrowing the gap between codes and supervision.
  • Notation: The binary code length is denoted by l.

3.5 Hashing Model

The hashing model jointly learns hash functions and binary codes from seen training samples with embedded label supervision. It uses kernelized features while retaining discrete codes and local structure, followed by alternating optimization.

  • Objective: The model learns hash functions from training samples X supervised by embedded label knowledge Y to produce binary codes for unseen-category data.
  • Joint Learning: It jointly controls hash functions and training binary codes because code quality affects hash-function quality.
  • Hash Function: The hash function uses a transformation matrix P and kernel mapping to address potential linear inseparability.
  • Binary Codes: The model keeps the discrete constraint on B to limit information loss in binary codes.
  • Local Structure: A local-structure term encourages similar original-space samples to receive similar Hamming-space codes.
  • Optimization: An efficient alternating algorithm is used to optimize the zero-shot hashing model.

3.6 Optimization

The optimization alternates updates for the hash function, binary codes, semantic alignment matrix, and mapping matrix. Binary codes retain their discrete constraint and are optimized with discrete coordinate descent until convergence.

  • Alternating optimization: The P subproblem incorporates the embedded features and graph Laplacian through a quadratic objective.The Laplacian is computed from the similarity structure, and its regularization term preserves local structure in the embedded feature space.
  • Discrete code update: Binary codes B are learned under a discrete constraint by updating each bit according to the sign of an expression involving the other l −1 bits.Cyclic coordinate descent generates the codes until the procedure converges.
  • Discrete code update: The DCC algorithm solves the binary-code subproblem by updating code rows while holding the remaining variables fixed.The optimization rewrites B row-wise and applies discrete coordinate descent to the resulting subproblem.
  • Matrix updates: With B, W, and P fixed, R is optimized through its subproblem, while W is obtained from a ridge regression with a closed-form solution.The alternating procedure then returns B, P, W, and R as the learned outputs.
  • Alternating optimization: The model is optimized by iteratively updating P, B, R, and W until the objective reaches an optimum.Algorithm 1 initializes the variables, maps data into the embedded feature space, constructs the Laplacian, and repeats the updates until no variable changes.

3.7 Algorithm Analysis

The alternating algorithm decreases the objective at every iteration and reaches a stable solution quickly in an ImageNet convergence study. Under the stated dimensional assumptions, its overall time cost is O(n).

  • Convergence: Each full iteration decreases the objective function, and the ImageNet study reports a sharp cost decline within 10 iterations.Training used 30,000 seen images, 1,000 anchors, and 64-bit codes; the cost started at roughly 30,000 before reaching a stable local minimum.
  • Computational complexity: The per-iteration updates cost O(m2n+nml+m3) for P, O(cl2+l2n) for B, O(c3) for R, and O(l2n+lnc+lc2+l3) for W.These costs correspond to the four alternating variable updates.
  • Computational complexity: O(n) is the overall time cost when m ≪n, l ≪n, c ≪n, and convergence occurs within fewer than 10 iterations.Matrix multiplication is the dominant operation and can be accelerated with parallel or distributed algorithms.

4. EXPERIMENT

Experiments evaluate zero-shot image retrieval across three datasets using Precision and MAP, comparing ZSH with supervised and unsupervised hashing baselines. ZSH generally achieves the strongest retrieval performance, while results vary with code length, unseen-category relatedness, and the amount of seen-category supervision.

  • Experimental setup: Experiments use CIFAR-10, ImageNet, and MIRFlickr, extracting fc-7 visual features and evaluating retrieval with Precision and MAP.MAP is reported over the top 5,000 retrieved samples, while Precision uses Hamming radius r ≤2.
  • Experimental setup: ZSH is compared with COSDISH, SDH, KSH, LFH, and the unsupervised IMH method using varying binary-code lengths.The experiments use code lengths of 16, 32, 64, 96, and 128 bits.
  • Overall comparison: Across CIFAR-10 code lengths, ZSH outperforms all other methods in MAP and achieves superior Precision in most cases.The authors attribute this to transferring semantic label relationships while preserving discrete and structural properties.
  • Code-length effects: MAP rises rapidly from 16 to 64 bits before slowing through 128 bits, whereas Precision deteriorates beyond 64 bits under the fixed Hamming-radius search.The paper presents code length as a trade-off between retrieval effectiveness and search difficulty.
  • Unseen-category effects: Zero-shot retrieval varies across unseen categories: bird performs best and automobile worst, with MAP positively related to average cosine similarity to seen categories.Dog and cat have relatively high similarity and retrieval performance, while airplane and automobile have relatively low values.
  • Supervision and dataset effects: Both MAP and Precision improve as the seen-category ratio increases, with a sharp gain from 0.1 to 0.3 followed by smaller improvements through 0.9.On ImageNet, ZSH also consistently outperforms competitors in most cases, while related-category retrieval reaches Precisionrelated values of 0.3262, 0.2636, and 0.2129 at 32, 64, and 96 bits.

5. CONCLUSION

The paper concludes that ZSH transfers supervised knowledge from seen to unseen categories by embedding labels, aligning semantics, and preserving structural and discrete code properties. It identifies multimodal semantic integration as future work for addressing domain difference and semantic shift more fundamentally.

  • ZSH transfers supervised knowledge from seen categories to unseen categories through semantic representations of independent 0/1-form labels.
  • The method rotates the embedding space to align supervised semantics more suitably with high-quality hash-code learning.
  • ZSH also preserves local structural properties and the discrete nature of hash codes, optimized with an iterative algorithm.
  • Future work will combine textual corpora and visual clues to improve label-semantic correlations and address domain difference and semantic shift.
Loading 1606.05032v1…