PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Semantic Ranking Based Hashing for Multi-Label Image Retrieval

Fang Zhao, Yongzhen Huang, Liang Wang, Tieniu Tan

arXiv:1501.06272v2cs.CVcs.LG

TL;DR

Multi-label image retrieval needs hashing methods that preserve more than binary similarity, because shared labels induce multilevel semantic relationships. DSRH jointly learns CNN representations and hash mappings under semantic ranking supervision, using a surrogate loss for optimization. Experiments report better ranking quality than state-of-the-art hashing methods, within a labeled binary-hashing formulation.

  • Problem

    Existing hashing methods mainly preserve binary semantic similarity, while multilevel relationships among multi-label images remain insufficiently addressed.

  • Method

    DSRH jointly learns CNN feature representations and binary hash mappings using ranking lists derived from shared labels and a triplet-based surrogate loss.

  • Results

    DSRH outperforms other state-of-the-art hashing methods in ranking quality on multi-label image retrieval evaluations.

  • Takeaways & Limitations

    Combining deep representation learning with multilevel semantic ranking supervision preserves the semantic structure of multi-label images for retrieval.

  • Takeaways & Limitations

    The formulation assumes labeled data and represents hash functions as mappings from D-dimensional inputs to binary codes.

Abstract

from arXiv · show

With the rapid growth of web images, hashing has received increasing interests in large scale image retrieval. Research efforts have been devoted to learning compact binary codes that preserve semantic similarity based on labels. However, most of these hashing methods are designed to handle simple binary similarity. The complex multilevel semantic structure of images associated with multiple labels have not yet been well explored. Here we propose a deep semantic ranking based method for learning hash functions that preserve multilevel semantic similarity between multi-label images. In our approach, deep convolutional neural network is incorporated into hash functions to jointly learn feature representations and mappings from them to hash codes, which avoids the limitation of semantic representation power of hand-crafted features. Meanwhile, a ranking list that encodes the multilevel similarity information is employed to guide the learning of such deep hash functions. An effective scheme based on surrogate loss is used to solve the intractable optimization problem of nonsmooth and multivariate ranking measures involved in the learning procedure. Experimental results show the superiority of our proposed approach over several state-of-the-art hashing methods in term of ranking evaluation metrics when tested on multi-label image datasets.

1. Introduction

The paper addresses the difficulty of preserving multilevel semantic similarity among multi-label images, where binary similarity and hand-crafted features are insufficient. It proposes DSRH, combining deep CNN hash functions with semantic ranking supervision and surrogate-loss optimization.

  • Binary hashing maps high-dimensional images to compact codes in Hamming space for efficient large-scale retrieval.
  • Multi-label images require multilevel similarity based on shared labels, which conventional binary-similarity hashing methods do not handle well.
  • Hand-crafted features can lose semantic information, limiting the standard pipeline of feature extraction followed by binary-code mapping.
  • DSRH jointly learns CNN feature representations and mappings to hash codes using ranking supervision derived from shared class labels.
  • A triplet-based surrogate ranking loss makes optimization of nonsmooth, multivariate ranking measures compatible with stochastic gradient descent.
  • The authors report state-of-the-art ranking evaluation performance for multi-label image retrieval.

2. Related Work

Related work progresses from data-dependent semantic hashing to ranking-based objectives and deep visual representation learning. DSRH builds on these directions by combining ranking supervision with deep hashing.

  • CCA-ITQ uses labels for dimensionality reduction and binarizes the result by minimizing quantization error.
  • Order-preserving hashing methods explicitly optimize similarity orders in feature or semantic spaces to preserve ranking structure.
  • Deep CNNs have shown strong representation-learning capability across visual tasks including classification, annotation, retrieval, and detection.
  • Prior CNN approaches use ranking losses for similarity learning or annotation, while deep hashing methods also incorporate deep models.

3. Our Method

DSRH learns deep hash functions that preserve multilevel semantic rankings among multi-label images. It combines CNN-based representations with ranking supervision and a surrogate-loss optimization scheme.

  • 3.1. Deep Hash Functions: DSRH uses CNNs to jointly learn image representations and mappings to compact binary hash codes, avoiding reliance on hand-crafted features.The architecture forwards resized images through five convolutional and two fully connected layers before generating the code.
  • 3.1. Deep Hash Functions: A skipping connection from FCa to the hash layer incorporates feature information biased toward visual appearance alongside deeper representations.The bypass is intended to reduce possible information loss when producing retrieval hash codes.
  • 3.2. Semantic Ranking Supervision: Semantic similarity levels are assigned from the number of labels shared with a query, producing a ground-truth ranking list for each query.Points sharing all query labels receive the highest level, while points sharing none receive level 0.
  • 3.2. Semantic Ranking Supervision: NDCG evaluates ranking consistency, but directly optimizing ranking measures is intractable because their losses are nonsmooth and multivariate.The method therefore uses a surrogate loss rather than directly optimizing the ranking criterion.
  • 3.3. Optimization with Surrogate Loss: The surrogate objective is a convex upper bound on pairwise ranking disagreement, expressed as weighted triplet losses with a Hamming-distance margin.The margin parameter controls the minimum distance gap between more- and less-relevant database points.
  • 3.3. Optimization with Surrogate Loss: Triplets receive larger weights when misordering them would cause a larger NDCG decline, emphasizing highly relevant results.Setting the adaptive weight to 1 recovers the unweighted surrogate loss.
  • 3.3. Optimization with Surrogate Loss: Stochastic gradient descent and CNN back-propagation optimize the surrogate objective, while balance and L2 penalties regularize the hash functions.The balance penalty encourages each bit to have mean-zero output over the training data.

4. Experiments

Experiments evaluate DSRH on MIRFLICKR-25K and NUS-WIDE using ranking metrics and comparisons with hashing methods under multiple feature settings. DSRH consistently performs best, while ablations show effects from adaptive weighting and the skipping-layer connection.

  • Experimental setup: Experiments use MIRFLICKR-25K and NUS-WIDE, comparing DSRH with unsupervised ITQ and SH and supervised CCA-ITQ and HDML using ranking measures.The evaluation reports NDCG, ACG, and weighted mAP; training uses mini-batch gradient descent with dropout and short ground-truth ranking lists.
  • Component analysis: Adaptive weighting improves top-100 NDCG and ACG but lowers averaged ranking performance by emphasizing more relevant database points.Connecting the first fully connected layer to the hash layer also improves performance by using information biased toward visual appearance.
  • Hand-crafted features: DSRH significantly outperforms hashing methods based on hand-crafted features across datasets, metrics, and tested code lengths.The comparison covers NDCG, ACG, and weighted mAP using various numbers of bits.
  • Pre-trained CNN features: DSRH retains better ranking performance than compared methods when they use pre-trained CNN activation features.The authors attribute this to jointly learning feature representations and hash codes with semantic supervision rather than hashing precomputed features.
  • Fine-tuned CNN features: DSRH achieves the best ranking performance with fine-tuned CNN features, supporting multilevel semantic ranking supervision for multi-label image structure.HDML performs worse than unsupervised ITQ in this setting, while ITQ performs nearly as well as CCA-ITQ and better on weighted mAP.
  • Component analysis: Applying compared hashing methods to concatenated activations from the last two CNN hidden layers performs worse, validating DSRH’s tight CNN–hash-function coupling.This experiment contrasts feature concatenation with DSRH’s integrated hash-function structure.

5. Conclusion

The paper concludes that DSRH uses CNNs and multilevel semantic ranking supervision to preserve multi-label semantic structure in binary codes. A triplet-based surrogate ranking loss enables effective stochastic optimization, and experiments show improved ranking quality over state-of-the-art hashing methods.

  • 5. Conclusion: DSRH jointly learns CNN feature representations and mappings to binary codes under listwise multilevel ranking supervision.The framework targets preservation of semantic structure in multi-label images.
  • 5. Conclusion: A triplet-based surrogate ranking loss makes the nonsmooth, multivariate ranking optimization amenable to stochastic gradient descent.The surrogate loss replaces direct optimization of ranking measures in the training procedure.
  • 5. Conclusion: Extensive experiments demonstrate that DSRH outperforms other state-of-the-art hashing methods in ranking quality.
Loading 1501.06272v2…