Source-linked AI summary
Self-Supervised Adversarial Hashing Networks for Cross-Modal Retrieval
Chao Li, Cheng Deng, Ning Li, Wei Liu, Xinbo Gao, Dacheng Tao
TL;DR
Cross-modal retrieval remains limited by the modality gap between heterogeneous representations. SSAH combines self-supervised semantic learning with two adversarial networks, and experiments on three benchmark datasets report state-of-the-art retrieval performance.
Problem
The modality gap between heterogeneous cross-modal representations limits retrieval accuracy and requires more abundant semantic relevance and accurate modality relationships.
Method
SSAH uses a self-supervised semantic network to generate multi-label semantic information and two adversarial networks to learn modality features and hash codes while preserving semantic relevance and distribution consistency.
Results
SSAH achieves state-of-the-art retrieval performance on three benchmark datasets and consistently achieves the best performance with vgg19 features.
Takeaways & Limitations
The learned semantic information preserves modality relationships in the common semantic space and Hamming space while adversarial learning addresses feature distribution differences.
Abstract
from arXiv · showhide
Thanks to the success of deep learning, cross-modal retrieval has made significant progress recently. However, there still remains a crucial bottleneck: how to bridge the modality gap to further enhance the retrieval accuracy. In this paper, we propose a self-supervised adversarial hashing (\textbf{SSAH}) approach, which lies among the early attempts to incorporate adversarial learning into cross-modal hashing in a self-supervised fashion. The primary contribution of this work is that two adversarial networks are leveraged to maximize the semantic correlation and consistency of the representations between different modalities. In addition, we harness a self-supervised semantic network to discover high-level semantic information in the form of multi-label annotations. Such information guides the feature learning process and preserves the modality relationships in both the common semantic space and the Hamming space. Extensive experiments carried out on three benchmark datasets validate that the proposed SSAH surpasses the state-of-the-art methods.
1. Introduction
Cross-modal hashing seeks efficient retrieval across heterogeneous modalities, but modality gaps and limited semantic supervision constrain existing methods. SSAH addresses these challenges with self-supervised semantic learning and adversarial networks, and experiments on three benchmark datasets report state-of-the-art performance.
- Cross-modal retrieval uses a query from one modality to find semantically similar instances in another, while hashing targets low storage cost and high query speed.
- Heterogeneous feature representations and distributions create a modality gap that cross-modal hashing must bridge by modeling semantic relevance in detail.
- Existing deep cross-modal hashing methods have limited semantic supervision, including direct reliance on single-class labels and short hash codes that can neutralize useful information.
- SSAH combines two adversarial networks with a self-supervised semantic network that discovers multi-label information to preserve semantic relevance and representation consistency across modalities.
- Experiments on three benchmark datasets show that SSAH significantly outperforms traditional and deep-learning-based state-of-the-art cross-modal hashing methods.
2. Related Work
Cross-modal hashing research includes unsupervised, supervised, and deep-learning approaches that preserve relationships across modalities using different sources of information. SSAH extends adversarial cross-modal retrieval with two adversarial networks for semantic relevance and representation consistency under learned semantic supervision.
- Unsupervised cross-modal hashing learns unified codes by discovering inter-modality and intramodality information from unlabeled training data.
- Supervised hashing exploits labels or semantic affinities to improve retrieval and preserve intra-class similarity or maximal cross-modal correlation.
- Deep cross-modal hashing uses end-to-end learning to preserve cross-modal similarities, including DCMH's negative log-likelihood objective.
- Unlike ACMR, which applies adversarial learning without hashing, SSAH uses two adversarial networks to jointly model modalities and capture semantic relevance and representation consistency.
3. Proposed SSAH
SSAH combines LabNet’s self-supervised multi-label semantic learning with ImgNet and TxtNet adversarial networks to preserve cross-modal relevance and representation consistency in semantic and Hamming spaces.
- Architecture: SSAH focuses on bimodal image-text retrieval using LabNet, ImgNet, and TxtNet as its three principal components.LabNet generates semantic features from multi-label annotations, while ImgNet and TxtNet process image and text modalities.
- Feature learning: ImgNet and TxtNet learn modality-specific hash functions while using LabNet’s semantic features to preserve relevance across modalities.The image and text objectives share a similar structure but use learned semantic information from LabNet to guide modality-feature learning.
- Self-supervised semantic generation: Multi-label annotations provide self-supervised semantic information that guides feature learning for both image and text modalities.Because each triplet contains image, text, and label information for the same instance, labels supervise modality-feature learning.
- Self-supervised semantic generation: LabNet preserves similarity relationships between semantic features and their corresponding hash codes through nonlinear projection.Its objective includes semantic-similarity preservation, hash-code similarity preservation, binarization approximation, and label classification losses.
- Adversarial learning: Two discriminators compare image or text features with LabNet semantic features to discover distribution differences and address the modality gap.Each discriminator receives modality features and label-derived semantic features, then performs binary modality classification.
- Optimization: Training iteratively updates network parameters by backpropagation and updates the binary code matrix with a sign function until convergence.The optimization uses image, text, and label sets, mini-batches, a learning rate, and a maximum iteration count.
4. Experiment
Experiments evaluate SSAH across three benchmark datasets using standard cross-modal retrieval protocols and comparisons with shallow and deep baselines. SSAH achieves stronger retrieval performance, benefits from its semantic and adversarial modules, and trains faster than DCMH.
- Datasets: The experiments use MIRFLICKR-25K, NUS-WIDE, and MS COCO, with BoW text representations and manually annotated semantic concepts.MIRFLICKR-25K uses 20,015 points and 1,386-dimensional text vectors; MS COCO uses 85,000 items and 2,000-dimensional text vectors.
- Evaluation and baselines: Evaluation uses MAP for Hamming ranking, precision-recall curves for hash lookup, and P@n curves for top-ranked precision.The compared methods include five shallow baselines and the deep method DCMH.
- Hamming ranking: SSAH exceeds shallow baselines by more than 10% MAP and DCMH by more than 5% MAP on MIRFLICKR-25K with CNN-F features.The comparison covers both image-to-text and text-to-image retrieval directions.
- Hamming ranking: With VGG19 features, SSAH consistently achieves the best performance and exceeds shallow baselines by more than 5% average MAP on MIRFLICKR-25K.The results are reported for image-to-text and text-to-image retrieval across three datasets.
- Hash lookup: SSAH significantly outperforms state-of-the-art competitors on 16-bit hash-code precision-recall curves across three benchmark datasets.The curves vary the Hamming radius from 0 to 16 in steps of 1.
- Ablation study: Removing the self-supervised semantic network or adversarial learning reduces performance, while the full design achieves more accurate results on MIRFLICKR-25K.The ablation compares SSAH with variants that remove or replace individual modules.
- Training efficiency: SSAH reduces training time by a factor of 10 compared with DCMH while learning modality correlations through LabNet, ImgNet, and TxtNet.The reported comparison examines MAP against training time.
- Comparison with ACMR: On the NUS-WIDE-10k setting used for comparison, SSAH significantly outperforms ACMR.The setting contains 10,000 randomly selected image-text pairs from the 10 largest NUS-WIDE categories.
5. Conclusion
SSAH is a deep hashing approach for cross-modal retrieval that combines self-supervised semantic learning with adversarial learning. Experiments report state-of-the-art retrieval performance on three benchmark datasets.
- SSAH is a deep hashing approach designed to address cross-modal retrieval.
- The method combines a self-supervised semantic network with multi-label information and adversarial learning.
- Adversarial learning maximizes semantic relevance and feature-distribution consistency between modalities.
- SSAH achieves state-of-the-art retrieval performance on three benchmark datasets.