PaperlayerView exact revision on arXiv

Source-linked AI summary

SSDH: Semi-supervised Deep Hashing for Large Scale Image Retrieval

Jian Zhang, Yuxin Peng

arXiv:1607.08477v3cs.CV

TL;DR

Existing deep hashing methods are mainly supervised and overlook the underlying structures of unlabeled data. SSDH combines semantic-similarity learning with structure-preserving embeddings and online graph construction. Experiments report the best search accuracy among eight state-of-the-art methods on five widely used datasets.

  • Problem

    Existing deep hashing methods are mainly supervised and use semantic information while ignoring the underlying structures of unlabeled data.

    Source: §I · p. 2

  • Method

    SSDH jointly learns image features and hash codes with a semi-supervised loss and online mini-batch graphs built using evolving features.

    Source: 2) Semi-supervised Embedding Term · p. 5, §V · p. 13

  • Results

    On five widely used datasets, SSDH reports the best search accuracy among eight state-of-the-art methods.

    Source: §I · p. 2

  • Takeaways & Limitations

    SSDH uses labeled and unlabeled data to capture semantic similarity and underlying data structure while learning hash codes and image features jointly.

    Source: §V · p. 13

Abstract

from arXiv · show

Hashing methods have been widely used for efficient similarity retrieval on large scale image database. Traditional hashing methods learn hash functions to generate binary codes from hand-crafted features, which achieve limited accuracy since the hand-crafted features cannot optimally represent the image content and preserve the semantic similarity. Recently, several deep hashing methods have shown better performance because the deep architectures generate more discriminative feature representations. However, these deep hashing methods are mainly designed for supervised scenarios, which only exploit the semantic similarity information, but ignore the underlying data structures. In this paper, we propose the semi-supervised deep hashing (SSDH) approach, to perform more effective hash function learning by simultaneously preserving semantic similarity and underlying data structures. The main contributions are as follows: (1) We propose a semi-supervised loss to jointly minimize the empirical error on labeled data, as well as the embedding error on both labeled and unlabeled data, which can preserve the semantic similarity and capture the meaningful neighbors on the underlying data structures for effective hashing. (2) A semi-supervised deep hashing network is designed to extensively exploit both labeled and unlabeled data, in which we propose an online graph construction method to benefit from the evolving deep features during training to better capture semantic neighbors. To the best of our knowledge, the proposed deep network is the first deep hashing method that can perform hash code learning and feature learning simultaneously in a semi-supervised fashion. Experimental results on 5 widely-used datasets show that our proposed approach outperforms the state-of-the-art hashing methods.

I. INTRODUCTION

SSDH addresses deep hashing’s reliance on labels and neglect of unlabeled-data structure by jointly learning hash codes and image features in a semi-supervised network. It adds a semi-supervised loss and online graph construction, and reports the best search accuracy among eight state-of-the-art methods on five widely used datasets.

II. RELATED WORK

  • Unsupervised hashing methods capture data properties such as neighborhood structure, whereas supervised methods use label-derived semantic information.Examples include SH and AGH for structure, and BRE and KSH for supervised similarity.Source: §I · p. 1

I. INTRODUCTION

  • Existing deep hashing methods are mainly supervised and overlook unlabeled-data structure, while SSDH learns hash codes and image features simultaneously in a semi-supervised fashion.The paper describes SSDH as the first deep hashing method to combine these learning tasks semi-supervisedly.Source: §I · p. 2
  • SSDH’s loss combines empirical error on labeled data with embedding error on labeled and unlabeled data to preserve semantic similarity and meaningful neighbors.The embedding component captures neighborhoods in underlying data structures.Source: §I · p. 2
  • Online graph construction builds neighborhood graphs within training mini-batches, avoiding the O(n^2) cost of offline construction and using evolving deep features.The smaller batch is suited to batch-based network training.Source: §I · p. 2
  • Across five widely used datasets, SSDH reports the best search accuracy among eight state-of-the-art methods.The introduction reports this overall comparison without giving metric values.Source: §I · p. 2

A. Convolutional Neural Network Hashing

  • CNNH learns approximate codes before fitting hash functions and image representations, so Stage 2 features cannot improve its Stage 1 codes.This two-stage separation limits hash-function learning.Source: A. Convolutional Neural Network Hashing · p. 3

B. Network in Network Hashing

NINH and BS-DRSCH are one-stage deep hashing methods that learn image features and hash codes while preserving supervised semantic similarity. Both rely on labeled data and leave unlabeled data structures unmodeled.

B. Network in Network Hashing

  • NINH uses a shared convolutional sub-network and divide-and-encode module to produce binary codes, with a triplet ranking loss guiding hash learning.Its optimization relaxes the binary-code constraint and replaces Hamming distance with Euclidean distance.Source: B. Network in Network Hashing · p. 3

C. Bit-scalable Deep Hashing

  • BS-DRSCH weights individual hash bits to flexibly manipulate code length and uses triplet tuples to preserve ranking and pairwise adjacency.Its convolutional layers extract features, while fully connected and tanh-like layers generate codes; an element-wise layer learns bit weights.Source: C. Bit-scalable Deep Hashing · p. 3
  • Both NINH and BS-DRSCH preserve label-based semantic similarity but ignore unlabeled data structures that can capture semantic neighborhoods.The paper identifies this as a limitation of both supervised methods.Source: C. Bit-scalable Deep Hashing · p. 3

III. SEMI-SUPERVISED DEEP HASHING

SSDH maps labeled and unlabeled images into hash codes while jointly learning image representations and hash functions. Its network combines these components with a semi-supervised loss that preserves semantic similarity and neighborhood structure.

A. Overview of proposed hashing framework

  • SSDH jointly preserves semantic similarity and underlying data structures by minimizing labeled empirical error and embedding error on labeled and unlabeled data.The framework couples representation learning, hash-code learning, and a semi-supervised loss.Source: A. Overview of proposed hashing… · p. 4

B. The Proposed Deep Network

  • The network uses CNN-F layers for image features, a sigmoid hash layer for q-bit codes, and a classification layer that predicts unlabeled-data pseudo labels.The outputs of fc7, fc8, and fc9 connect to the semi-supervised loss in an end-to-end network.Source: B. The Proposed Deep Network · p. 4
  • Continuous hash outputs are thresholded at 0.5 to obtain binary codes, but optimization relaxes binary codes to continuous values.The hash layer maps fc7 features into [0, 1]^q using learned weights and a bias.Source: B. The Proposed Deep Network · p. 4

C. Semi-supervised Deep Hashing Learning

The SSDH loss combines supervised ranking with semi-supervised embedding and pseudo-label terms. The ranking term uses labeled triplets, while the embedding term captures structure across labeled and unlabeled data.

C. Semi-supervised Deep Hashing Learning

  • The three loss terms preserve labeled semantic similarity, capture underlying data structure, and encourage decision boundaries in low-density regions.The embedding term is complementary to the pseudo-label term.Source: C. Semi-supervised Deep Hashing Learning · p. 5

1) Supervised Ranking Term:

  • The supervised ranking term samples labeled triplets to preserve relative semantic similarity provided by annotations.For dissimilar pairs, the desired distance exceeds the similar-pair distance by at least margin mt.Source: 1) Supervised Ranking Term · p. 5
  • Because labeled images are costly to collect, SSDH adds an embedding term over labeled and unlabeled data, using the manifold assumption that points on the same manifold likely share labels.Graph-based embedding is used to capture the underlying data structures.Source: 2) Semi-supervised Embedding Term · p. 5

2) Semi-supervised Embedding Term:

SSDH constructs neighborhood graphs online within each mini-batch using features from its deep network. A contrastive embedding term then encourages graph neighbors to be close and non-neighbors to be separated.

2) Semi-supervised Embedding Term:

  • Online mini-batch graph construction avoids building a full-data k-NN graph, whose time and memory costs are O(n^2).The method computes each batch's k-nearest neighbors using the network's deep features.Source: 2) Semi-supervised Embedding Term · p. 5
  • The graph links labeled and unlabeled images but omits labeled-labeled edges, since supervised ranking already models their similarity.Batch size r is much smaller than n, and evolving deep features can capture semantic neighbors more accurately during training.Source: 2) Semi-supervised Embedding Term · p. 5
  • The embedding term uses contrastive regularization to bring graph neighbors closer and keep non-neighbors apart by margin mp.It measures embedding error on both labeled and unlabeled data to preserve neighborhood structure.Source: 2) Semi-supervised Embedding Term · pp. 5–6

3) Pseudo-label Term:

The pseudo-label term uses the classification branch to derive label-based similarity for unlabeled data, complementing the neighborhood graph; the semi-supervised loss combines this term with supervised ranking and embedding terms.

3) Pseudo-label Term:

  • The classification branch predicts pseudo-labels for unlabeled images, using maximum-probability classes or labels from k-nearest labeled neighbors for multi-label data.A contrastive loss models semantic similarity between the predicted pseudo-labels.Source: 3) Pseudo-label Term · p. 6

4) The Semi-supervised Loss:

  • The semi-supervised loss combines supervised ranking, semi-supervised embedding, and pseudo-label terms, while sampling only part of graph-derived image pairs to balance neighbors and non-neighbors.The loss uses λ and µ as balance parameters.Source: 4) The Semi-supervised Loss · p. 6

D. Network Training

Training uses mini-batches containing labeled and unlabeled images, jointly updating hash-code and classification branches; the trained model encodes new images for Hamming-distance retrieval.

D. Network Training

  • Each training iteration updates the network from a mini-batch containing labeled and unlabeled images.The batch includes m labeled images and r total images.Source: D. Network Training · p. 6, L , YB · p. 6

L , YB

  • Joint training uses cross-entropy on labeled data for the classification branch and SGD with back-propagation for the hash-code branch.The branches generate hash codes and predict pseudo-labels, respectively.Source: L , YB · p. 6
  • After training, the model encodes new images as binary hash codes, enabling retrieval by fast Hamming-distance ranking.The model is trained until the loss converges or a preset maximum iteration count is reached.Source: L , YB · p. 7

IV. EXPERIMENTS

The experiments compare hashing methods across four image datasets using retrieval metrics and curve-based evaluations, alongside further baseline, transfer-protocol, and large-scale ImageNet experiments.

IV. EXPERIMENTS

  • Experiments cover four datasets and eight state-of-the-art methods spanning unsupervised, supervised, and semi-supervised hashing.The study also evaluates loss terms, λ and µ, online graph construction, transfer testing, and ImageNet.Source: §IV · p. 7

A. Datasets and evaluation metrics

  • The datasets are CIFAR-10, MNIST, NUS-WIDE, and MIRFLICKR, with randomly selected query sets and labeled training subsets.NUS-WIDE evaluation uses its 21 most frequent concepts, totaling 166,047 images.Source: A. Datasets and evaluation metrics · pp. 7–8
  • Evaluation uses MAP, Precision-Recall curves, Precision@topk, and Precision@top500 to assess retrieval performance.MAP averages the average precision for each query; Fig. 2 presents the three curve-based measures across datasets and code lengths.Source: A. Datasets and evaluation metrics · p. 8, Figure 2 · p. 8

B. Implementation details

SSDH achieves higher MAP than supervised deep-hashing baselines on the evaluated datasets, while the experiments also compare input features and describe the implementation setup.

B. Implementation details

  • SSDH uses CNN-F initialization, 128-image mini-batches, and λ = 0.1 and µ = 0.1; deep baselines share the CNN-F base structure and initialization.Traditional baselines are also evaluated with hand-crafted and CNN-F deep features.Source: B. Implementation details · pp. 8–9

C. Experiment Results and Analysis

  • 81.0% MAP on CIFAR10 for SSDH exceeds DRSCH* (73.2%), NINH* (67.2%), and CNNH* (56.0%); SSDH also outperforms supervised deep methods across four datasets and metrics.The comparison is reported in Table III and includes NINH*, DRSCH*, and CNNH*.Source: Table III · p. 9, C. Experiment Results and Analysis · p. 9
  • On CIFAR10, SDH-CNNF reaches 49.0% average MAP versus 32.2% for SDH using hand-crafted features.The paper reports that traditional hashing methods using deep features outperform the identical methods using hand-crafted features.Source: C. Experiment Results and Analysis · p. 9
  • SSDH outperforms compared methods on Precision-Recall, Precision@topk, and Precision@top500 curves, including across code lengths; CIFAR10 Precision@top500 exceeds 80% at every tested length.The curves cover four datasets and use 48-bit codes for the first two measures.Source: C. Experiment Results and Analysis · p. 9

D. Comparison of Computation Time

The loss-term comparisons support the contributions of the semi-supervised embedding and pseudo-label terms, while SSDH’s computation time is comparable to other deep hashing methods. Qualitative retrieval examples also report SSDH as the best-performing method.

D. Comparison of Computation Time

  • SSDH’s computation time is comparable to six other deep hashing methods, while DRSCH is relatively slow because it uses weighted Hamming distance.Hash generation after training is a matrix multiplication, and the compared methods use Hamming distance for search.Source: D. Comparison of Computation Time · p. 10

1) Evaluation of the Loss Function:

  • Across all four datasets and code lengths, SSDH_base beats NINH*, while full SSDH beats SSDH_base, supporting embedding-term effectiveness and complementarity with pseudo-labeling.The comparisons evaluate the semi-supervised embedding term and the combination of embedding and pseudo-label terms.Source: 1) Evaluation of the Loss Function · p. 10
  • The Fig. 3 examples report SSDH as achieving the best top-10 Hamming-ranking retrieval results on NUS-WIDE and CIFAR10.Blue rectangles mark query images and red rectangles mark wrong retrievals.Source: Figure 3 · p. 10, 1) Evaluation of the Loss Function · p. 10

2) Evaluation of Parameters:

The parameter study reports little sensitivity to the tested loss weights, and the online graph outperforms an offline graph. Graph and pseudo-label accuracy rise during training, while retrieval MAP is stable for smaller neighborhood sizes and declines for larger ones.

2) Evaluation of Parameters:

3) Evaluation of Online Graph Construction:

  • Online graph construction yields better retrieval results than offline construction at every tested hash-code length on CIFAR10.The online graph uses evolving deep features; the offline graph is fixed from pre-trained CNN-F features.Source: 3) Evaluation of Online Graph… · p. 10, Table VI · p. 11

4) Evaluation of the Impact of k:

  • On CIFAR10 with 48-bit codes, MAP is stable for k < 10 and decreases for k > 10.The authors attribute the decline at larger k to reduced k-nearest-neighbor accuracy; stability below 10 supports robustness.Source: 4) Evaluation of the Impact of k · pp. 11–12

3) Evaluation of Online Graph Construction:

  • Neighborhood-graph and pseudo-label accuracy both exceed 80% within 2,000 iterations, while retrieval accuracy rises alongside them during training.The authors relate the improving accuracies to evolving deep features and increasingly meaningful semantic neighbors.Source: 3) Evaluation of Online Graph… · p. 11, Figure 8 · p. 12

F. Experiment Results on Transfer Type Testing Protocols

Under transfer testing with unseen categories, SSDH achieves the best retrieval results among the compared methods; it also leads on a large-scale ImageNet evaluation. The conclusion describes a semi-supervised network and loss that use labeled and unlabeled data, and identifies further graph and embedding improvements as future work.

F. Experiment Results on Transfer Type Testing Protocols

  • With 75% of categories known during training and the remaining 25% used for evaluation, SSDH achieves the best retrieval results under transfer testing.The protocol uses the held-out categories for queries and includes a database distraction set without ground-truth images.Source: F. Experiment Results on Transfer Type… · p. 12, Table VIII · p. 13

G. Experiment Results on Large Scale ImageNet Dataset

  • On ImageNet under the transfer protocol, SSDH achieves the best reported top-500 retrieval precision; MAP is omitted because its evaluation cost is high.The dataset split contains 499,071 training images, 166,513 query images, and 665,583 database images.Source: Table IX · p. 13, G. Experiment Results on Large Scale… · p. 13

V. CONCLUSION

  • The conclusion describes a network that learns image features and hash codes semi-supervisedly, with ranking, embedding, and pseudo-label terms targeting labeled and unlabeled data.The authors report effectiveness compared with eight state-of-the-art methods.Source: §V · p. 13
  • Future work will explore improved semi-supervised embedding and graph-construction methods, and an unsupervised extension using clustering for virtual image labels.Source: §V · p. 13

Related papers

Loading 1607.08477v3…