Source-linked AI summary
Deep Cross-Modal Hashing
Qing-Yuan Jiang, Wu-Jun Li
TL;DR
Existing cross-modal hashing methods largely depend on hand-crafted features and generally separate feature extraction from hash-code learning, while deep CMH integration had not appeared. DCMH combines feature learning and hash-code learning in an end-to-end framework with one deep neural network per modality, and experiments on two datasets report state-of-the-art performance against other baselines. The paper’s scope includes a focus on two modalities and does not focus on designing different neural-network architectures.
Problem
Existing CMH methods largely use hand-crafted features independently of hash-code learning, and no deep CMH method had integrated both processes in one framework.
Method
DCMH is an end-to-end CMH framework that uses one deep neural network per modality to learn features from scratch while integrating hash-code learning.
Results
Experiments on two datasets show that DCMH outperforms other baselines and achieves state-of-the-art performance in cross-modal retrieval.
Takeaways & Limitations
DCMH demonstrates a cross-modal retrieval framework that simultaneously performs feature learning and hash-code learning.
Takeaways & Limitations
The paper focuses on two modalities and treats neural-network architecture design as outside its main scope, leaving other architectures for future study.
Abstract
from arXiv · showhide
Due to its low storage cost and fast query speed, cross-modal hashing (CMH) has been widely used for similarity search in multimedia retrieval applications. However, almost all existing CMH methods are based on hand-crafted features which might not be optimally compatible with the hash-code learning procedure. As a result, existing CMH methods with handcrafted features may not achieve satisfactory performance. In this paper, we propose a novel cross-modal hashing method, called deep crossmodal hashing (DCMH), by integrating feature learning and hash-code learning into the same framework. DCMH is an end-to-end learning framework with deep neural networks, one for each modality, to perform feature learning from scratch. Experiments on two real datasets with text-image modalities show that DCMH can outperform other baselines to achieve the state-of-the-art performance in cross-modal retrieval applications.
1. Introduction
Cross-modal hashing addresses multimodal retrieval with compact, fast binary representations, but existing methods largely rely on hand-crafted features and lack integrated deep feature and hash-code learning. DCMH proposes an end-to-end framework and reports state-of-the-art cross-modal retrieval performance.
- Motivation: Hashing reduces storage cost and supports constant or sub-linear search for approximate nearest-neighbor retrieval.Binary codes preserve similarity in Hamming space while enabling indexed search.
- Motivation: Cross-modal hashing is suited to multimodal data when queries or database items may provide different modalities.Multimodal examples include images and associated text such as tags.
- Research gap: Existing CMH methods generally use hand-crafted features whose extraction is independent of hash-code learning.The paper identifies this separation as a shortcoming of prior methods.
- Research gap: No prior deep CMH method had simultaneously learned features and hash codes within one framework, according to the authors.This motivates integrating both learning stages with deep neural networks.
- Contribution: DCMH uses one deep neural network per modality in an end-to-end framework to learn features from scratch and integrate hash-code learning.The method is presented as a novel CMH approach for cross-modal retrieval.
- Results: Experiments on real text-image datasets found that DCMH outperformed other baselines and achieved state-of-the-art cross-modal retrieval performance.The introduction reports this result without providing numerical values.
2. Problem Definition
The problem is to learn two binary hash functions that preserve cross-modal semantic similarity between image and text data. The formulation assumes paired training modalities, while allowing one- or two-modality queries after training.
- Setting: The paper focuses on two modalities, although the proposed method can be adapted to more than two modalities.The two-modality setting is used for the paper’s text-image illustration.
- Problem formulation: The cross-modal similarity matrix S marks image-text pairs as similar or dissimilar, typically using shared class labels.Sij = 1 denotes similarity and Sij = 0 denotes dissimilarity.
- Problem formulation: Cross-modal hashing learns two hash functions mapping image and text features to c-bit binary codes.The codes are constrained to binary values and should preserve cross-modal similarity.
- Problem formulation: Similar image-text pairs should have small Hamming distance, whereas dissimilar pairs should have large Hamming distance.This is the target relationship between the learned binary codes and S.
- Setting: Training assumes both modalities are observed for each training point, but trained models can generate codes for one- or two-modality queries and database points.This matches the intended cross-modal retrieval setting.
3. Deep Cross-Modal Hashing
The paper introduces DCMH as a deep cross-modal hashing method and presents its model formulation and learning algorithm.
- Deep Cross-Modal Hashing: DCMH is presented as a deep cross-modal hashing method.The section announces the method’s model formulation and learning algorithm.
- Deep Cross-Modal Hashing: The method section covers DCMH’s model formulation.
- Deep Cross-Modal Hashing: The method section covers DCMH’s learning algorithm.
3.1. Model
DCMH integrates feature learning and hash-code learning in one end-to-end framework, using separate deep networks for images and text. Its objective learns network parameters and binary codes jointly while preserving cross-modal similarity.
- Feature learning part: DCMH uses two deep neural networks, one for image features and one for text features, within an end-to-end framework.The image network is a CNN, while the text network receives bag-of-words vectors.
- Feature learning part: The image network contains eight layers, including five convolutional and three fully-connected layers, with its eighth layer producing learned image features.The first seven layers use ReLU, while the eighth uses the identity function.
- Feature learning part: The text modality is represented with bag-of-words vectors and processed by a three-layer fully-connected network.The first two layers use ReLU and the third uses the identity function.
- Scope: The paper focuses on demonstrating an end-to-end deep cross-modal hashing framework, leaving alternative neural-network designs for future study.The authors explicitly state that designing different neural networks is outside this paper’s focus.
- Hash-code learning part: DCMH’s objective learns image-network parameters, text-network parameters, and binary hash codes from the same objective function.The framework therefore integrates feature learning and hash-code learning rather than treating them as separate stages.
- Hash-code learning part: The objective’s similarity term encourages large image-text feature inner products for similar pairs and small inner products for dissimilar pairs.The learned continuous features serve as surrogates for binary codes that preserve the cross-modal similarity matrix.
- Hash-code learning part: The model balances hash-code bits across training points and sets the two modalities’ training codes to be identical.The bit-balancing constraint aims to make the information provided by each bit more effective.
3.2. Learning
DCMH learns its image network, text network, and binary codes through alternating optimization. Each update fixes the other variables, uses mini-batch backpropagation for network parameters, and then optimizes the codes.
- Alternating optimization: DCMH uses an alternating learning strategy that optimizes θx, θy, and B one variable group at a time.Each parameter update keeps the other parameters fixed.
- Image-network update: With θy and B fixed, the image-network parameter θx is updated by stochastic-gradient backpropagation on sampled mini-batches.The image features are computed by forward propagation before the gradient and parameter update.
- Text-network update: With B and θx fixed, the text-network parameter θy is likewise learned with stochastic-gradient backpropagation.The update computes a gradient for each sampled text point using the chain rule.
- Algorithm: Algorithm 1 alternates image and text mini-batch updates before optimizing B and repeating until a fixed iteration count.The stated mini-batch sizes are Nx = Ny = 128.
- Binary-code update: When the network parameters are fixed, each binary code Bij is assigned the same sign as its corresponding continuous value Vij.This yields the code update B = sign(V).
3.3. Out-of-Sample Extension
DCMH generates hash codes for unseen query or database points by forward propagation from whichever modality is available. This supports cross-modal search between single-modality queries and opposite-modality database points.
- Code generation: For an unseen point with an image modality, DCMH generates its hash code by applying forward propagation to the learned image network.The procedure is described for an image query modality xq.
- Code generation: For an unseen point with only text, DCMH similarly generates a hash code from the learned text network.The text-only case uses the corresponding text modality yq.
- Cross-modal retrieval: DCMH supports cross-modal search when query points have one modality and database points have the other modality.The model can generate codes for either modality after training.
4. Experiment
Experiments evaluate DCMH against five cross-modal hashing baselines on two text-image datasets using Hamming ranking and hash lookup. Across hand-crafted and CNN-F feature settings, DCMH generally achieves the strongest reported retrieval performance, while its hash-lookup results favor recall and F-measure.
- Datasets and protocol: Experiments use MIRFLICKR-25K and NUS-WIDE text-image datasets, with semantic-label overlap defining ground-truth neighbors.MIRFLICKR-25K contributes 20,015 selected points, while NUS-WIDE contributes 186,577 selected text-image pairs.
- Datasets and protocol: DCMH is compared with SePH, STMH, SCM, CMFH, and CCA using Hamming ranking and hash lookup.The evaluation reports MAP for Hamming ranking and precision, recall, and F-measure across Hamming radii for hash lookup.
- Hamming ranking: DCMH outperforms all baselines with hand-crafted features on MIRFLICKR-25K and NUS-WIDE in MAP.The results are reported in Tables 3 and 4.
- Hamming ranking: With CNN-F features, DCMH outperforms all baselines except SePH in most cases and beats SePH with linear features in all cases.SePH remains stronger in the NUS-WIDE image-to-text case when using kernel features, while DCMH wins most other comparisons.
- Hash lookup: With 16-bit codes and hand-crafted-feature baselines, DCMH dramatically outperforms the baselines in precision-recall curves on the two datasets.Figure 2 evaluates image-to-text and other cross-modal query/database directions.
- Hash lookup: For 16-bit hash lookup, DCMH achieves the best recall and F-measure at Hamming radii 0, 1, and 2 across all reported cases.DCMH returns nearly 580 points, including 487 ground-truth neighbors, in the MIRFLICKR-25K image-to-text radius-0 example, whereas STMH returns 3 points.
5. Conclusion
DCMH is an end-to-end cross-modal hashing framework that learns features from scratch while learning hash codes. Experiments on two datasets show state-of-the-art cross-modal retrieval performance over other baselines.
- DCMH performs feature learning from scratch within an end-to-end cross-modal hashing framework.
- DCMH simultaneously learns modality features and hash codes in the same framework.
- Experiments on two datasets show that DCMH outperforms other baselines in cross-modal retrieval applications.