PaperlayerView exact revision on arXiv

Source-linked AI summary

SCH-GAN: Semi-supervised Cross-modal Hashing by Generative Adversarial Network

Jian Zhang, Yuxin Peng, Mingkuan Yuan

arXiv:1802.02488v1cs.CV

TL;DR

Cross-modal hashing must bridge differences between modalities, while supervised methods depend on costly labeled data and often leave unlabeled examples unused. SCH-GAN adversarially selects margin-near unlabeled examples to challenge a discriminator, and experiments report the best retrieval accuracy among compared methods on three datasets.

  • Problem

    Cross-modal hashing must bridge the heterogeneity gap, while supervised approaches rely on large labeled cross-modal training sets and make limited use of unlabeled data.

    Source: §I · pp. 1–2

  • Method

    SCH-GAN uses a generator to select margin examples from unlabeled data and a discriminator to distinguish them from true positive pairs, trained adversarially with reinforcement learning.

    Source: §I · p. 2, 2) Optimizing generative model · p. 6

  • Results

    SCH-GAN achieves the best retrieval accuracy across Wikipedia, NUSWIDE, and MIRFlickr; on NUSWIDE image-query-text, average MAP rises from 0.505 (CCQ) to 0.730.

    Source: E. Comparison with state-of-the-art… · p. 10

  • Takeaways & Limitations

    The learned hashing functions reflect both labeled-data semantics and unlabeled-data distributions.

    Source: §I · p. 2

Abstract

from arXiv · show

Cross-modal hashing aims to map heterogeneous multimedia data into a common Hamming space, which can realize fast and flexible retrieval across different modalities. Supervised cross-modal hashing methods have achieved considerable progress by incorporating semantic side information. However, they mainly have two limitations: (1) Heavily rely on large-scale labeled cross-modal training data which are labor intensive and hard to obtain. (2) Ignore the rich information contained in the large amount of unlabeled data across different modalities, especially the margin examples that are easily to be incorrectly retrieved, which can help to model the correlations. To address these problems, in this paper we propose a novel Semi-supervised Cross-Modal Hashing approach by Generative Adversarial Network (SCH-GAN). We aim to take advantage of GAN's ability for modeling data distributions to promote cross-modal hashing learning in an adversarial way. The main contributions can be summarized as follows: (1) We propose a novel generative adversarial network for cross-modal hashing. In our proposed SCH-GAN, the generative model tries to select margin examples of one modality from unlabeled data when giving a query of another modality. While the discriminative model tries to distinguish the selected examples and true positive examples of the query. These two models play a minimax game so that the generative model can promote the hashing performance of discriminative model. (2) We propose a reinforcement learning based algorithm to drive the training of proposed SCH-GAN. The generative model takes the correlation score predicted by discriminative model as a reward, and tries to select the examples close to the margin to promote discriminative model by maximizing the margin between positive and negative data. Experiments on 3 widely-used datasets verify the effectiveness of our proposed approach.

I. INTRODUCTION

Cross-modal hashing maps different modalities into a shared Hamming space for fast retrieval, but the heterogeneity gap and dependence on costly labels limit learning. SCH-GAN uses unlabeled margin examples in an adversarial approach to improve cross-modal hashing.

A. Cross-modal Hashing

  • Cross-modal hashing bridges heterogeneous modalities in a common Hamming space, enabling retrieval across modalities rather than only within one modality.The paper contrasts this flexibility with single-modality retrieval.Source: §I · p. 1

II. RELATED WORKS

  • Prior cross-modal hashing includes unsupervised methods that learn from data distributions, supervised methods that use semantic labels, and deep-learning-based methods.The related-work discussion names representative methods in each category.Source: §I · p. 2, A. Cross-modal Hashing · p. 3

I. INTRODUCTION

  • Supervised cross-modal hashing benefits from semantic labels, but large-scale cross-modal datasets are labor-intensive to label and semi-supervised approaches remain little explored.The paper identifies exploiting informative unlabeled examples as the central challenge.Source: §I · p. 2, A. Cross-modal Hashing · p. 3
  • SCH-GAN selects unlabeled margin examples to challenge a discriminator that distinguishes them from true positive pairs, jointly learning from unlabeled distributions and labeled semantics.The two models optimize each other through a minimax game.Source: §I · p. 2
  • Experiments report that SCH-GAN outperforms eight state-of-the-art methods on Wikipedia, NUSWIDE, and MIRFlickr.The paper presents this as evidence of the approach’s effectiveness.Source: §I · p. 2

B. Generative Adversarial Network

SCH-GAN applies adversarial learning to cross-modal hashing: a generator selects informative unlabeled examples, while a discriminator learns semantic ranking. The approach uses labeled and unlabeled data to learn mappings into a shared Hamming space.

B. Generative Adversarial Network

A. Notation

  • Cross-modal hashing learns image and text mappings into a common Hamming space, where semantically similar items can be retrieved across modalities using Hamming distance.The retrieval database also serves as the training set, and the query set can contain either modality.Source: A. Notation · pp. 4–5
  • The training database combines labeled data with a much larger unlabeled set, which the method exploits to promote hashing learning.The notation specifies that the unlabeled set size m is much greater than n, the labeled set size.Source: A. Notation · pp. 4–5

B. Network structure

The network uses paired image and text pathways for feature representation and hashing. Its generator selects margin-near unlabeled examples, while the discriminator judges generated and true pairs.

B. Network structure

  • The two-pathway generative model uses fixed CNN features for images and bag-of-words features for text, followed by fully connected layers for common-space mapping and hashing.The CNN parameters remain fixed during training because the focus is hashing-function learning.Source: B. Network structure · p. 5
  • The hashing layer maps intermediate features to continuous codes in [0, 1]^q, then thresholds them at 0.5 to obtain binary codes.The final fully connected layer has dimension q, the hash code length.Source: B. Network structure · p. 5
  • Binary codes are relaxed to continuous-valued hash codes during optimization, and the generator selects informative unlabeled examples near margins for a query.The resulting mappings support fast cross-modal Hamming-distance comparison.Source: B. Network structure · p. 5
  • The discriminator shares the generator’s two-pathway structure and distinguishes selected pairs from true relevant pairs sampled from labeled data.Its inputs include generated pairs and true pairs.Source: B. Network structure · p. 5

C. Objective function

The objective couples a generator that selects cross-modal candidates from unlabeled data with a discriminator that distinguishes them from labeled relevant pairs. Their minimax interaction trains the discriminator to separate true positives from margin examples.

  • The generator selects relevant unlabeled data across modalities, while the discriminator distinguishes those selections from true relevant pairs in a minimax game.The discriminator uses pairs sampled by semantic labels alongside generator-selected pairs.Source: C. Objective function · p. 5
  • The image-query/text-candidate objective is defined symmetrically to the text-query/image-candidate objective.Source: C. Objective function · p. 5
  • The generator assigns selection probabilities from cross-modal hash distances, favoring images closer to a text query.The objective uses a softmax over candidate images; smaller image–text distances receive larger probabilities.Source: C. Objective function · p. 5
  • The discriminator’s triplet-ranking score separates a labeled positive from a generator-selected image by margin 1.The paper sets the margin parameter to 1 and also seeks to preserve ranking relations among labeled data.Source: C. Objective function · p. 6

D. Optimization

Optimization alternates between training the discriminator and generator, keeping the other model’s parameters fixed at each step.

  • Training alternates between the two models, fixing the generator during discriminator updates and vice versa.The section introduces separate optimization algorithms for the two models.Source: D. Optimization · p. 6

1) Optimizing discriminative model:

The discriminator is trained against labeled and generated pairs with the generator fixed; the generator is then updated with the discriminator fixed, using policy-gradient reinforcement learning to select margin examples.

1) Optimizing discriminative model:

  • With the generator fixed, the discriminator trains on generated and labeled query–candidate pairs to maximize correct distinction between them.The generated pairs come from the previous iteration, and true pairs are sampled from labeled data.Source: 1) Optimizing discriminative model · p. 6

2) Optimizing generative model:

  • During generator optimization, the discriminator remains fixed while the generator minimizes the shared objective.Source: 2) Optimizing generative model · p. 6
  • Because candidate selection is discrete, the generator is updated with policy-gradient reinforcement learning rather than stochastic gradient descent.The discriminator’s correlation score serves as the reward, encouraging selection of examples close to the margin.Source: 2) Optimizing generative model · p. 6

E. Cross-modal retrieval by learned discriminative model

After training, the discriminative model performs cross-modal retrieval by encoding a query into a binary hash code and comparing it with database codes using Hamming distance.

  • Retrieval uses the trained discriminator’s hash codes and fast Hamming-distance comparisons between the query and database items.A query of either modality is first encoded as a binary hash code.Source: E. Cross-modal retrieval by learned… · p. 7

IV. EXPERIMENTS

The experiments compare SCH-GAN with eight state-of-the-art methods on three widely used cross-modal retrieval datasets. The datasets vary in size and labeled-training setup, with only a fraction of NUSWIDE and MIRFlickr used for supervised training.

IV. EXPERIMENTS

  • Experiments compare SCH-GAN with 8 state-of-the-art methods on Wikipedia, NUSWIDE, and MIRFlickr; NUSWIDE uses 186,557 selected images, with 1% as query data, while MIRFlickr contains 25,000 images.Source: §IV · p. 7, A. Dataset · p. 7

A. Dataset

  • Wikipedia uses 2,173 training pairs and 693 test pairs, with the training set also serving as the retrieval database.It contains 2,866 image-text pairs from 10 categories.Source: A. Dataset · p. 7
  • NUSWIDE uses 1% of its selected 186,557 images for queries and randomly samples 5,000 images for supervised training.The dataset has 81 concepts; the experiments select its 10 most common concepts.Source: A. Dataset · p. 7
  • MIRFlickr uses 5% of its 25,000 images for queries and randomly samples 5,000 images for supervised training.The images have one or more of 24 semantic labels.Source: A. Dataset · p. 7

B. Retrieval tasks and evaluation metrics

The experiments evaluate image-to-text and text-to-image retrieval using Hamming ranking and three retrieval metrics. MAP is the mean of query-level average precisions, while precision-recall and topK-precision assess ranked retrieval at different levels.

B. Retrieval tasks and evaluation metrics

  • Image queries retrieve semantically similar texts, and text queries retrieve semantically similar images from the retrieval database.Hamming ranking orders results by Hamming distance, with semantic neighbors expected near the top.Source: B. Retrieval tasks and evaluation… · pp. 7–8
  • Tables I–III report MAP scores for both retrieval tasks across different hash-code lengths on Wikipedia, NUSWIDE, and MIRFlickr.The supplied passages identify the tables’ scope but do not provide their cell values.Source: Table I · p. 8, Table II · p. 8, Table III · p. 8
  • MAP, precision-recall curves, and topK-precision measure retrieval effectiveness across the two tasks.MAP averages query-level average precision; precision-recall reports precision at specified recall, and topK-precision measures precision across different retrieval counts.Source: B. Retrieval tasks and evaluation… · p. 8

C. Comparison methods

The comparison includes unsupervised, supervised, and deep-learning cross-modal hashing baselines. Traditional methods receive the same image and text features, while deep methods use the specified VGGNet and BoW inputs.

C. Comparison methods

  • Unsupervised baselines include CVH, which models intra- and inter-view similarities, PDH, which preserves code predictability, CMFH, which uses collective factorization, and CCQ, which combines correlation mappings with composite quantization.CVH, PDH, CMFH, and CCQ.Source: C. Comparison methods · p. 9
  • Supervised baselines include CMSSH, which frames hashing as classification, SCM, which preserves label-based semantic similarity, and SePH, which approximates a semantic probability distribution in Hamming space.CMSSH, SCM, and SePH.Source: C. Comparison methods · p. 9
  • DCMH jointly learns features and hashing functions; traditional methods use 4096-dimensional VGGNet image features and 1000-dimensional BoW text features for comparison.Deep methods use the same VGGNet base network for images and the same text-pathway settings.Source: C. Comparison methods · p. 9

D. Implementation details

SCH-GAN uses fixed VGGNet image features, BoW text features, and fully connected hashing layers, then trains its generator and discriminator iteratively. It achieves the best reported retrieval accuracy among the compared methods on three datasets and both retrieval directions.

1) Data processing:

  • The image pathway uses fixed 19-layer VGGNet features, while the text pathway uses 1000-dimensional BoW features.The experiments focus on adversarial hashing training rather than updating VGGNet parameters.Source: 1) Data processing · p. 9

2) Details of network:

  • The hashing layers comprise a 4096-dimensional fully connected intermediate layer and a sigmoid hashing layer sized to the hash code length.The sigmoid constrains the hashing-layer output to [0, 1].Source: 2) Details of network · p. 9

E. Comparison with state-of-the-art methods

  • On the three datasets and both retrieval directions, SCH-GAN achieves the best accuracy in the reported 64-bit topK-precision and precision-recall curves.The figure rows represent image-query-text and text-query-image; columns represent Wikipedia, NUSWIDE, and MIRFlickr.Source: Figure 5 · p. 10, E. Comparison with state-of-the-art… · p. 11

3) Training details:

  • Training uses batches of 64 per modality, generates 20 pairs per query, and alternates one-epoch discriminator and generator steps.The learning rate starts at 0.01 and decreases tenfold every two epochs; compared methods use their authors’ implementations and recommended settings.Source: 3) Training details · p. 10

E. Comparison with state-of-the-art methods

  • 0.730 versus 0.715 MAP on NUSWIDE image-to-text, and 0.758 versus 0.654 on text-to-image, are reported for SCH-GAN versus best supervised SePH.SCH-GAN also reports 0.730 versus 0.505 for best unsupervised CCQ on image-to-text, and 0.758 versus 0.490 on text-to-image; it is reported best across the three datasets.Source: E. Comparison with state-of-the-art… · p. 10

F. Baseline experiments

SCH-GAN outperforms the non-adversarial Dis baseline on all three datasets, while retrieval accuracy rises after generator updates during adversarial training. The conclusion reports evaluation against eight state-of-the-art methods on three datasets and identifies video, audio, and unsupervised settings as future extensions.

F. Baseline experiments

  • Two baseline experiments assess model retrieval during training and compare SCH-GAN with an approach lacking adversarial training.The experiments are intended to provide insight into adversarial training and verify its effectiveness.Source: F. Baseline experiments · p. 11

1) Performance of adversarial training:

  • During adversarial training, discriminative-model retrieval accuracy increases after generative-model updates.The paper attributes this to the generator selecting more informative examples for the discriminator.Source: 1) Performance of adversarial training · p. 11

2) Comparison with baseline approach:

V. CONCLUSION

  • The conclusion reports that SCH-GAN was compared with eight state-of-the-art methods on three widely used datasets.The approach selects unlabeled margin examples and uses reinforcement-learning-based training, with the generator taking the discriminator's correlation score as a reward.Source: §V · p. 11
  • Future work would extend the framework to unsupervised learning and to video and audio modalities.The stated aim is to exploit complex correlations among multiple modalities.Source: §V · p. 11

Related papers

Loading 1802.02488v1…