Source-linked AI summary
Unsupervised Generative Adversarial Cross-modal Hashing
Jian Zhang, Yuxin Peng, Mingkuan Yuan
TL;DR
Existing unsupervised cross-modal hashing methods preserve inter- and intra-modal correlations but overlook manifold structure that can identify meaningful cross-modal neighbors. UGACH combines a graph-based correlation model with adversarial unsupervised learning to exploit that structure, and experiments report effectiveness across two datasets against six state-of-the-art methods. The paper’s demonstrated scope is limited to two-modality retrieval, with extension to multiple modalities left for future work.
Problem
Unsupervised cross-modal hashing avoids intensive labeling but existing methods largely overlook underlying cross-modal manifold structure useful for meaningful nearest neighbors.
Method
UGACH combines a graph-based correlation approach with a generative adversarial network that selects informative cross-modal data and trains hashing functions adversarially.
Results
UGACH achieves the best retrieval accuracy on two datasets, including average MAP scores of 0.624 for image→text and 0.625 for text→image on NUS-WIDE.
Takeaways & Limitations
Cross-modal data within the same manifold can be assigned smaller Hamming distances, supporting more accurate cross-modal retrieval.
Takeaways & Limitations
The reported approach is scoped to current cross-modal settings, while retrieval across multiple modalities and scenarios such as image captioning remain future work.
Abstract
from arXiv · showhide
Cross-modal hashing aims to map heterogeneous multimedia data into a common Hamming space, which can realize fast and flexible retrieval across different modalities. Unsupervised cross-modal hashing is more flexible and applicable than supervised methods, since no intensive labeling work is involved. However, existing unsupervised methods learn hashing functions by preserving inter and intra correlations, while ignoring the underlying manifold structure across different modalities, which is extremely helpful to capture meaningful nearest neighbors of different modalities for cross-modal retrieval. To address the above problem, in this paper we propose an Unsupervised Generative Adversarial Cross-modal Hashing approach (UGACH), which makes full use of GAN's ability for unsupervised representation learning to exploit the underlying manifold structure of cross-modal data. The main contributions can be summarized as follows: (1) We propose a generative adversarial network to model cross-modal hashing in an unsupervised fashion. In the proposed UGACH, given a data of one modality, the generative model tries to fit the distribution over the manifold structure, and select informative data of another modality to challenge the discriminative model. The discriminative model learns to distinguish the generated data and the true positive data sampled from correlation graph to achieve better retrieval accuracy. These two models are trained in an adversarial way to improve each other and promote hashing function learning. (2) We propose a correlation graph based approach to capture the underlying manifold structure across different modalities, so that data of different modalities but within the same manifold can have smaller Hamming distance and promote retrieval accuracy. Extensive experiments compared with 6 state-of-the-art methods verify the effectiveness of our proposed approach.
Introduction
Multimedia retrieval requires efficient search across heterogeneous data, motivating cross-modal hashing into a shared Hamming space. Existing unsupervised methods avoid labeling costs but largely preserve correlations while overlooking cross-modal manifold structure.
- Retrieval motivation: Hashing maps high-dimensional representations to short binary codes, enabling efficient Hamming-distance computation and lower storage use.Binary codes support fast bit operations for large-scale retrieval.
- Retrieval motivation: Single-modality retrieval cannot directly support searches such as retrieving images with textual sentences describing their content.Cross-modal hashing addresses the need to retrieve across different multimedia modalities.
- Cross-modal challenge: Cross-modal hashing must bridge the heterogeneous gap because similarity between different modalities cannot be measured directly.Existing approaches include traditional unsupervised and supervised methods, as well as DNN-based methods.
- Existing approaches: Unsupervised methods project modalities into a common Hamming space by preserving inter-modal and intra-modal correlations.Representative approaches include IMH, CVH, PDH, CMFH, and CCQ.
- Existing approaches: Supervised methods use semantic labels, but collecting those labels requires substantial labor and cost, making unsupervised retrieval more flexible for real-world applications.The distinction between supervised and unsupervised methods is whether semantic information is leveraged.
- Proposed direction: UGACH combines unsupervised GAN-based learning with graph-based correlation modeling to exploit cross-modal manifold structure.The proposed method uses a generative model to select informative cross-modal data and a discriminative model to distinguish generated from true positive data.
The Proposed Approach
UGACH combines feature extraction, a generative model, and a discriminative model to learn unsupervised cross-modal hash codes. A correlation graph captures cross-modal manifold structure, while adversarial training makes generated pairs challenge discrimination and supports retrieval.
- Framework: UGACH extracts image and text features, then uses generative and discriminative models to learn a cross-modal hashing model.The trained discriminative model is used for cross-modal retrieval through produced hash codes.
- Feature and Hashing Layers: The common representation layer converts modality-specific features into a measurable shared space before hashing.The hashing layer then maps common representations to continuous values and thresholded binary codes; relaxed values are used during training.
- Generative Model: The generative model selects informative unpaired data from another modality according to a learned manifold distribution to form challenging pairs.For example, an image query can guide selection of a relevant text instance from the retrieval database.
- Discriminative Model: A correlation graph captures cross-modal manifold structure and supplies manifold pairs whose members should have small Hamming distance.Pairwise information lets corresponding instances transfer same-manifold relations across modalities.
- Discriminative Model: The discriminative model distinguishes graph-sampled manifold pairs from generated pairs using relevance scores and triplet ranking.The ranking objective makes the manifold-pair distance smaller than the generated-pair distance by margin m, set to 1.
- Adversarial Learning: UGACH trains its generative and discriminative models through a minimax game, using reinforcement-learning updates because selection from unlabeled data is discrete.The discriminator maximizes the objective, while the generator minimizes it and fits the manifold distribution.
Experiments
UGACH is evaluated on two cross-modal retrieval tasks and compared with unsupervised, supervised, and baseline methods using MAP, precision-recall, and topK-precision metrics. It achieves the best reported retrieval accuracy across both datasets, with ablations supporting contributions from adversarial training and the correlation graph.
- Datasets and comparisons: Experiments use NUS-WIDE and MIRFlickr to evaluate image→text and text→image retrieval.The comparison includes 6 state-of-the-art methods and 2 baseline methods.
- Ablation design: The baseline trains the discriminative model with paired positives, whereas Baseline-GAN adds adversarial training to that baseline.These baselines isolate the effects of adversarial training and the correlation graph.
- Evaluation metrics: MAP is evaluated across four hash-code lengths, while precision-recall and topK-precision curves use 128-bit codes.The figures report both retrieval directions across the two datasets.
- Main results: UGACH achieves the best retrieval accuracy on both datasets and maintains the highest NUS-WIDE average MAP scores: 0.624 for image→text and 0.625 for text→image.On NUS-WIDE, it improves over CCQ from 0.505 to 0.624 and from 0.494 to 0.625, respectively.
- Main results: On both retrieval tasks, UGACH achieves the best accuracy among compared unsupervised methods and exceeds supervised methods on most evaluation metrics.This pattern is reported for the topK-precision and precision-recall evaluations with 128-bit codes.
- Ablation results: Baseline-GAN improves over the baseline by 0.037 and 0.034 on NUS-WIDE, while UGACH further improves over Baseline-GAN by 0.025 and 0.016.The authors report similar trends on MIRFlickr.
Conclusion
UGACH combines adversarial learning with graph-based correlation learning to exploit cross-modal manifold structure. Experiments on two datasets against six state-of-the-art methods support its effectiveness, while future work targets more modalities and scenarios.
- UGACH: UGACH uses a generative adversarial network to exploit the underlying manifold structure of cross-modal data.The generative model fits manifold distributions and challenges the discriminative model with informative cross-modal data.
- UGACH: Its graph-based correlation learning makes same-manifold data from different modalities have smaller Hamming distances.The stated goal is to promote cross-modal retrieval accuracy.
- Evidence: Experiments against 6 state-of-the-art methods on 2 widely-used datasets verify the effectiveness of UGACH.
- Future work: Future work extends the framework to multiple modalities and scenarios such as image captioning.The proposed extensions include image, text, video, and audio retrieval.