PaperlayerView exact revision on arXiv

Source-linked AI summary

Cycle-Consistent Deep Generative Hashing for Cross-Modal Retrieval

Lin Wu, Yang Wang, Ling Shao

arXiv:1804.11013v2cs.CV

TL;DR

Cross-modal hashing needs to preserve semantic relationships across heterogeneous modalities, but paired training samples are not always available. CYC-DGH addresses this gap with adversarially trained cross-modal hash mappings, cycle consistency, and generative input reconstruction. Experiments report advances over state-of-the-art methods on image-to-text and text-to-image retrieval across three benchmarks.

  • Problem

    Cross-modal hashing must handle modality heterogeneity and semantic gaps while learning hash functions without paired training correspondence.

  • Method

    CYC-DGH jointly learns cross-modal generative hash functions through adversarial and cycle-consistency losses, while reconstructing inputs from binary codes.

  • Results

    CYC-DGH advances the state of the art on image-to-text and text-to-image retrieval tasks across three benchmarks.

  • Takeaways & Limitations

    The framework provides a generative cross-modal hashing approach that learns without paired samples and couples binary-code learning with input regeneration.

Abstract

from arXiv · show

In this paper, we propose a novel deep generative approach to cross-modal retrieval to learn hash functions in the absence of paired training samples through the cycle consistency loss. Our proposed approach employs adversarial training scheme to lean a couple of hash functions enabling translation between modalities while assuming the underlying semantic relationship. To induce the hash codes with semantics to the input-output pair, cycle consistency loss is further proposed upon the adversarial training to strengthen the correlations between inputs and corresponding outputs. Our approach is generative to learn hash functions such that the learned hash codes can maximally correlate each input-output correspondence, meanwhile can also regenerate the inputs so as to minimize the information loss. The learning to hash embedding is thus performed to jointly optimize the parameters of the hash functions across modalities as well as the associated generative models. Extensive experiments on a variety of large-scale cross-modal data sets demonstrate that our proposed method achieves better retrieval results than the state-of-the-arts.

I. INTRODUCTION

Cross-modal hashing must bridge heterogeneous modalities and semantic gaps, while many existing deep approaches rely on labeled aligned pairs. CYC-DGH addresses this by combining adversarial translation, cycle consistency, and generative reconstruction to learn hash functions without paired correspondence.

  • Motivation: Large-scale cross-modal retrieval uses compact binary codes to preserve semantic similarity across heterogeneous modalities.The task seeks efficient nearest-neighbor search across images and texts despite modality heterogeneity and the gap between low-level features and high-level semantics.
  • Limitations of Existing Methods: Existing shallow hashing methods learn single projections into binary codes, which the paper describes as suboptimal for modality-specific optimization and high-level semantic representation.The cited methods project examples into an independent semantic embedding space, but image low-level descriptions may inadequately express semantics.
  • Limitations of Existing Methods: Deep cross-modal hashing models jointly learn feature transformations and hash functions, but supervised approaches use semantic labels and labeled aligned pairs.The paper identifies paired correspondence as a key source of supervision for preserving cross-modal semantics.
  • CYC-DGH: CYC-DGH learns generative hash mappings between image and text modalities without paired training correspondence by exploiting adversarial learning and cycle consistency.The mappings G and F translate between modalities while the cycle constraint encourages each composition to recover its input.
  • CYC-DGH: The generative mapping jointly learns binary codes and regenerates inputs from those codes, aiming to minimize information loss during hash embedding.The mapping is decomposed into binary-code learning and input regeneration, with cycle training returning translated inputs to their original domains.

B. Contributions

The paper contributes a cycle-consistent adversarial hashing framework for heterogeneous modalities, designed to learn cross-modal hash functions without paired samples. It also couples binary-code learning with input regeneration and positions the method against prior cross-modal retrieval approaches.

  • B. Contributions: CYC-DGH introduces cross-modal adversarial training to model data distributions and address heterogeneity between modalities.The framework applies adversarial learning across modalities rather than only learning a shared representation.
  • B. Contributions: Cycle consistency is added to cross-modal adversarial training so hash functions can be learned without paired training samples.The cycle constraint encourages reverse mappings to recover the original inputs while translating between modalities.
  • B. Contributions: Deep generative models jointly learn binary codes and regenerate inputs from those codes to reduce information loss in hash embeddings.This generative view couples hashing with reconstruction rather than treating code learning as projection alone.
  • Related Work: Generative adversarial networks have also been used for heterogeneous-data modeling and synthetic image generation, but those goals differ from learning hash functions without paired samples.The paper distinguishes its objective from prior GAN-based common-representation and text-conditioned image-generation methods.
  • Related Work: Prior cross-modal hashing methods include unsupervised co-occurrence-based approaches, supervised semantic methods, and deep models trained from labeled pairs.The paper contrasts CYC-DGH with common-subspace, hashing, and end-to-end deep frameworks that rely on pair or label information.

C. Image-to-Image Translation

CYC-DGH extends unpaired image-to-image translation into cross-modal hashing by combining bidirectional mappings, adversarial discrimination, binary-code generation, decoding, and cycle consistency.

  • Image-to-image translation maps one representation of a scene into another, with unpaired methods addressing the dependence on aligned training examples.
  • CYC-DGH learns bidirectional mappings between modalities without paired input-output examples, using separate image and text sample sets.
  • Each mapping combines binary-code learning with a reverse process that regenerates inputs from the learned codes.
  • The objective jointly includes adversarial losses, cycle-consistency losses, and reconstruction loss from binary codes.
  • Adversarial discriminators distinguish real target-modality samples from translated samples for both mapping directions.

B. Cycle-Consistency Loss

The cycle-consistency design uses cross-modal generative decoding to connect binary codes with inputs and regulate unpaired translations. Its Gaussian reconstruction model is intended to preserve local neighborhood structure relevant to retrieval.

  • Cycle consistency regulates adversarially learned mappings so translations can be learned across domains without paired correspondence.
  • Cycle-consistent translation returns an input toward its original form through the generative Hamming space formed by encoding and decoding functions.
  • The mappings encode each modality into binary codes and decode those codes into inputs from the other modality.
  • The generative model uses a Gaussian distribution for generating an input from its binary code and a multivariate Bernoulli prior over hash codes.
  • Minimizing Gaussian reconstruction error is described as a surrogate for preserving Euclidean neighborhoods when the Frobenius norm of U is bounded.

D. Training Objective

The training objective combines adversarial translation, cycle consistency, and generative reconstruction to learn cross-modal hash functions while preserving input information.

  • Objective formulation: The full objective combines adversarial losses in both translation directions with cycle-consistency and generative terms.The two adversarial components are followed by λLcyc(G, F ), a KL-divergence term, and reconstruction-related objective L(Θ∗; x∗).
  • Cycle-consistent reconstruction: The model jointly trains F ◦G : Xu →Xu and G ◦F : Xv →Xv as cross-modal auto-encoders.Each cycle maps an input through an intermediate translation into the other modality before regenerating the original input.
  • Parameterization: The generative parameters include modality-specific model parameters and encoding parameters W ∗ from the hash function.Θ∗ = {W ∗, U ∗, ρ∗, β∗}, with U ∗, ρ∗, and β∗ belonging to p(x∗, h∗).
  • Gradient estimation: A stochastic neuron re-parameterizes Bernoulli binary variables to approximate gradients through the discrete hash codes.The method uses a distributional-derivative approximation because standard stochastic gradients with respect to W depend on stochastic binary variables.

E. Training Details

Implementation uses modality-specific deep networks, a PatchGAN discriminator, and stabilization techniques for adversarial training.

  • Network architecture: Text features are processed by fully connected networks whose dimensions vary across COCO, IAPR TC-12, and Wiki.The reported architectures are [1000 →500 →200], [11500 →500 →200], and [10 →100 →200], respectively.
  • Network architecture: The image generator follows a style-transfer architecture, while the discriminator is a 70 × 70 PatchGAN.The patch discriminator classifies overlapping image patches and can handle arbitrarily sized images.
  • Training stabilization: Training stability is improved by replacing the GAN negative log-likelihood with least-squares loss and updating the discriminator using generated-image history.These strategies target network oscillation and adversarial-training instability.

IV. EXPERIMENTS

Experiments evaluate CYC-DGH against state-of-the-art methods on three widely used cross-modal retrieval benchmarks with different dataset scales and annotations.

  • Experimental scope: The evaluation compares CYC-DGH with state-of-the-art methods on three benchmark datasets.The experiments are designed to assess retrieval efficiency across multiple large-scale cross-modal settings.
  • Datasets: Microsoft COCO provides 82,081 training images after removing images without category information, with at least five sentence annotations per image.The dataset uses 80 frequent categories as ground-truth labels.
  • Datasets: IAPR TC-12 contributes 18,673 training images selected using 22 frequent concept tags from a 20,000-image collection.Each image has at least one sentence annotation and category annotations based on 275 concepts.
  • Datasets: Wiki contains 2,866 documents, each pairing one image with text of at least 70 words across 10 semantic classes.The data are split into 75% database documents and 25% query samples.

B. Competitors and Evaluation Setup •

The evaluation covers diverse cross-modal hashing baselines and measures image-to-text and text-to-image retrieval using mean average precision.

  • Competitors: Competitors include TUCH, CMDVH, DVSH, CorrAE, CMNN, CAH, DCMH, and HashGAN.These methods represent translation, variational, metric-based, autoencoder, similarity-preserving, and adversarial hashing approaches.
  • Competitors: DCMH uses an end-to-end deep framework with negative log likelihood to preserve similarity between same-class real-valued representations.Its formulation is presented as a supervised deep hashing baseline.
  • Variant evaluation: Table I reports FCN-scores for CYC-DGH variants evaluated on Microsoft COCO using 64-bit codes to regenerate images.The comparison is among model variants under a fixed dataset and code length.
  • Evaluation metrics: The evaluation performs both image-to-text and text-to-image retrieval across the datasets.Queries search texts using images or images using texts, respectively.
  • Evaluation metrics: Mean average precision summarizes the average precision of all queries, with precision computed over ranked retrieved instances.Average precision uses relevance indicators at each retrieval rank.

C. Ablation Studies

The ablation analysis examines the proposed adversarial and cycle-consistency losses, while reconstruction experiments compare CYC-DGH with ITQ across three data sets. The evaluation also considers reconstruction error as the number of training examples increases.

  • Loss ablations: The study compares the full objective against adversarial-only and cycle-consistency-only ablations.The ablation isolates the contributions of L_GAN and L_cyc.
  • Reconstruction loss: CYC-DGH reconstructs inputs from binary codes using a generative model, with regenerated inputs estimated by maximizing p(x|h).The reconstruction error is computed between the regenerated input and the original input.
  • Reconstruction loss: ITQ minimizes binary quantization loss and uses an encoding matrix with binary codes defined as B = sign(XW).The paper relates this objective to L2 reconstruction loss when feature-vector magnitudes match the binary-code radius.
  • Efficiency: Training-time comparisons for Microsoft-COCO and IAPR TC-12 are reported in Tables II and III.The supplied passages identify the table comparisons but provide no numerical training-time values.
  • Reconstruction loss: Figure 4 reports average L2 reconstruction loss against the number of examples seen during training on three data sets.The comparison includes CYC-DGH and ITQ, with image reconstruction evaluated using parameters learned from the text domain for ITQ.

2) Comparison to State-of-the-art Cross-Modal Hashing Methods:

CYC-DGH is evaluated against unsupervised and supervised shallow cross-modal hashing methods on two retrieval directions across three benchmarks. The reported comparisons include mAP and precision-recall behavior at 32 bits.

  • Compared methods: The evaluation compares CYC-DGH with unsupervised CVH and LSSH and supervised SCM and SePH methods.Image retrieval uses pretrained CNN-F FC7 image features for comparison with non-deep-learning methods.
  • Evaluation: The two tasks are image-to-text and text-to-image retrieval, evaluated using mAP and precision-recall curves.Figure 5 presents precision-recall curves at 32 bits on Microsoft COCO, IAPR TC-12, and Wiki.
  • Results: CYC-DGH provides the best performance compared with the shallow cross-modal hashing methods.The supplied result passage states this comparison without reporting numerical values.
  • Scope: The paired-sample requirement limits the practical application of cross-modal hashing methods that use aligned training pairs.This limitation is stated in the comparison discussion.

3) Comparison to Deep Cross-Modal Hashing Methods:

CYC-DGH is compared with deep cross-modal hashing competitors and evaluated through mAP, precision-recall, and precision@top-R curves. Across the supplied experiments, it achieves the strongest reported retrieval performance over three benchmarks while using generative, cycle-consistent hashing.

  • Comparison to deep methods: CYC-DGH achieves the best mAP results across different hashing-bit settings and three data sets against recent deep competitors.The comparison is reported in Table VI, although the supplied passages contain no mAP values.
  • Comparison to deep methods: The compared deep methods learn nonlinear cross-modal relationships or shared binary codes, but modality-specific hash functions remain limited to their specific modalities.The discussion contrasts these designs with CYC-DGH’s cross-modal objective.
  • Precision-recall evaluation: At 32 bits, CYC-DGH achieves the best precision-recall performance for both retrieval tasks at all recall levels across three benchmarks.The paper attributes this to cycle consistency, adversarial training, and input regeneration from binary codes.
  • Precision@top-R evaluation: CYC-DGH outperforms all competitors in precision@top-R curves and remains effective as the number of retrieved items increases.Figure 6 plots precision against the number R of top-retrieved results on three data sets at 32 bits.
  • Overall conclusion: The conclusion reports advances over state-of-the-art image-to-text and text-to-image retrieval across three benchmarks.The reported evidence includes competitor comparisons and ablation studies.
Loading 1804.11013v2…