Source-linked AI summary
Comprehensive Graph-conditional Similarity Preserving Network for Unsupervised Cross-modal Hashing
Jun Yu, Hao Zhou, Yibing Zhan, Dacheng Tao
TL;DR
UCMH methods can produce inaccurate similarities when pairwise cross-modal features miss complex relationships, limiting retrieval in Hamming space. DGCPN incorporates graph-neighbor coherence, three complementary similarity types, and half-real, half-binary optimization. Experiments on three public datasets demonstrate its capability for UCMH, while the graph-coherence computation is limited for efficiency.
Problem
UCMH similarities mainly based on pairwise cross-modal features can be inaccurate because those features insufficiently describe complex data relationships.
Method
DGCPN combines graph-neighbor coherence with coexistent and intra- and inter-modality consistency losses, then uses half-real, half-binary optimization for hashing.
Results
Experiments on three publicly available UCMH datasets compared with state-of-the-art methods demonstrate DGCPN’s capability for UCMH.
Takeaways & Limitations
DGCPN explores relationships between data and their neighbors to obtain more accurate similarities for UCMH.
Takeaways & Limitations
For computational efficiency, DGCPN imposes a limitation when calculating graph-neighbor coherence because the full conditional calculation is complicated.
Abstract
from arXiv · showhide
Unsupervised cross-modal hashing (UCMH) has become a hot topic recently. Current UCMH focuses on exploring data similarities. However, current UCMH methods calculate the similarity between two data, mainly relying on the two data's cross-modal features. These methods suffer from inaccurate similarity problems that result in a suboptimal retrieval Hamming space, because the cross-modal features between the data are not sufficient to describe the complex data relationships, such as situations where two data have different feature representations but share the inherent concepts. In this paper, we devise a deep graph-neighbor coherence preserving network (DGCPN). Specifically, DGCPN stems from graph models and explores graph-neighbor coherence by consolidating the information between data and their neighbors. DGCPN regulates comprehensive similarity preserving losses by exploiting three types of data similarities (i.e., the graph-neighbor coherence, the coexistent similarity, and the intra- and inter-modality consistency) and designs a half-real and half-binary optimization strategy to reduce the quantization errors during hashing. Essentially, DGCPN addresses the inaccurate similarity problem by exploring and exploiting the data's intrinsic relationships in a graph. We conduct extensive experiments on three public UCMH datasets. The experimental results demonstrate the superiority of DGCPN, e.g., by improving the mean average precision from 0.722 to 0.751 on MIRFlickr-25K using 64-bit hashing codes to retrieve texts from images. We will release the source code package and the trained model on https://github.com/Atmegal/DGCPN.
Introduction
Cross-modal hashing seeks efficient retrieval in compact Hamming space, but UCMH similarities based mainly on pairwise cross-modal features can miss intricate relationships. DGCPN addresses this with graph-neighbor coherence, complementary similarity-preserving losses, and half-real, half-binary optimization.
- Cross-modal hashing uses compact Hamming space to reduce storage and improve retrieval efficiency for multimedia data.
- Unsupervised CMH avoids unavailable or costly semantic labels by using whether heterogeneous data are coexistent.
- Pairwise cross-modal features can miss intricate semantic relationships, such as the connection between rose and wedding inferred from shared neighbors.
- Previous similarity designs only slightly outperform image-modal similarities, whereas graph-neighbor coherence has the best consistency with label-derived real similarities.
- DGCPN preserves graph-neighbor coherence, coexistent similarity, and intra- and inter-modality consistency through comprehensive similarity-preserving losses.
- Its half-real and half-binary optimization reduces value and similarity gaps between real-valued and Hamming spaces, lowering quantization errors.
- Extensive experiments on three public datasets show improved performance over state-of-the-art methods, demonstrating DGCPN’s competitiveness.
Related UCMH Work
Related UCMH work evolved from shallow, handcrafted-feature methods toward deep neural networks and continuous similarity modeling. These methods preserve or approximate similarities while addressing limitations of earlier discrete formulations.
- Earlier UCMH methods learned binary codes by preserving graph consistency, collective matrix factorization, or multigraph-regularized matrix factorization.
- Many previous methods relied on human-crafted features and shallow models, which limited retrieval performance.
- Later methods introduced deep neural networks, including deep autoencoders and coupled or multi-pathway generative adversarial networks.
- Some UCMH methods learned real-valued representations and converted them to binary codes with a Sign function.
- Earlier discrete similarity models assigned binary relations from feature distances but ignored data pairs with intermediate distances.
DGCPN
DGCPN preserves cross-modal similarities by combining graph-neighbor coherence with coexistent similarity and modality consistency, while jointly reducing value and similarity gaps between relaxed and binary representations.
- Framework: DGCPN uses feature-extracting subnetworks followed by similarity-preserving subnetworks that produce relaxed real values and binary codes for images and texts.The framework supports batch-wise training and can use alternative feature-extraction backbones.
- Graph-neighbor Coherence: Graph-neighbor coherence models data as graph nodes containing coexistent image-text pairs and estimates similarity from relationships between nodes and their neighbors.The graph formulation transforms data-similarity calculation into node-similarity calculation and uses a conditional-probability model.
- Graph-neighbor Coherence: DGCPN computes graph-neighbor coherence using cross-modal feature distances, k-nearest neighbors, conditional probabilities, and a transformed similarity range for training.The distance combines image- and text-feature cosine similarities, while γ and β adjust the resulting formulation.
- Comprehensive Similarity Preserving Losses: Comprehensive similarity-preserving losses regulate graph-coherence preservation, coexistent similarity preservation, and intra- and inter-modality consistency preservation.The final objective combines these losses with λ1 and λ2 controlling their relative significance.
- Half-real and Half-binary Optimization Strategy: The half-real and half-binary strategy reduces both value gaps and similarity gaps between relaxed real-valued representations and Hamming-space codes.It preserves similarities using one real value and one binary code because optimization with pure binary codes is difficult.
Experiments
DGCPN is evaluated on three public datasets using I2T and T2I retrieval tasks, MAP, and comparisons with nine UCMH methods. It achieves the best reported performance and benefits from its graph-neighbor coherence, comprehensive similarity losses, and half-real/half-binary optimization strategy.
- Datasets and Evaluation Metrics: DGCPN is evaluated on Wikipedia, MIRFlickr-25K, and NUS-WIDE using image-to-text and text-to-image retrieval tasks.The experiments use MAP as the retrieval metric.
- Performance Comparison: DGCPN obtains the best performance on all datasets regardless of the evaluation criteria.It is compared with nine UCMH methods, including five deep models.
- Performance Comparison: 4.4%, 4.2%, and 4.0% are DGCPN’s MIRFlickr-25K I2T improvements over DSAH at 16-, 32-, and 64-bit code lengths.For T2I, the corresponding improvements are 3.1%, 3.9%, and 2.9%.
- Ablation Experiments: DGCPN-NoPD outperforms DGCPN-OlPD, while DGCPN performs best, indicating the importance of graph-neighbor coherence and pairwise distances.The ablations use MIRFlickr-25K and NUS-WIDE as testbeds.
- Ablation Experiments: DGCPN outperforms variants using only graph-coherence losses or omitting consistency losses, supporting comprehensive similarity preservation.The reported results attribute the benefit to coexistent similarities and intra- and inter-modality consistency alleviating graph-coherence errors.
- Ablation Experiments: DGCPN outperforms variants without the half-real and half-binary strategy, showing the strategy improves the retrieval space by reducing similarity gaps.The compared variants use adjusted Tanh or a hashing function to reduce value gaps between real values and binary codes.
Conclusion
DGCPN combines graph-neighbor coherence, complementary similarity-preserving losses, and half-real/half-binary optimization for unsupervised cross-modal hashing. Experiments on three public datasets demonstrate its capability compared with state-of-the-art methods.
- DGCPN proposes graph-neighbor coherence to model relationships between data and their neighbors for more accurate similarity estimation.
- Comprehensive similarity-preserving losses regulate three complementary similarity types for robust similarity-preserving learning.
- Half-real and half-binary optimization reduces value and similarity gaps between real-valued and Hamming spaces, targeting better retrieval performance.
- Experiments on three publicly available UCMH datasets compared with state-of-the-art methods demonstrate DGCPN's capability for UCMH.