Source-linked AI summary
HashNet: Deep Learning to Hash by Continuation
Zhangjie Cao, Mingsheng Long, Jianmin Wang, Philip S. Yu
TL;DR
Existing deep hashing methods struggle to train sign-activated networks and therefore learn continuous codes before separate binarization, while similarity data are often imbalanced. HashNet uses continuation with weighted pairwise likelihood to learn exactly binary codes end to end. It achieves state-of-the-art retrieval performance and substantially improves average MAP over comparison methods on standard datasets.
Problem
Sign activation is needed for exact binary codes but makes back-propagation infeasible, while sparse similarity data create imbalance that weakens similarity-preserving learning.
Method
HashNet progressively changes tanh(βz) into sign activation during training and uses weighted pairwise maximum likelihood to learn from imbalanced similarity relationships.
Results
HashNet generates exactly binary hash codes and substantially outperforms comparison methods, including absolute average-MAP boosts of 14.6%, 3.7%, and 2.9% over DHN on ImageNet, NUS-WIDE, and MS COCO.
Takeaways & Limitations
HashNet provides an end-to-end deep hashing framework that jointly preserves similarity information and generates exactly binary codes for multimedia retrieval.
Abstract
from arXiv · showhide
Learning to hash has been widely applied to approximate nearest neighbor search for large-scale multimedia retrieval, due to its computation efficiency and retrieval quality. Deep learning to hash, which improves retrieval quality by end-to-end representation learning and hash encoding, has received increasing attention recently. Subject to the ill-posed gradient difficulty in the optimization with sign activations, existing deep learning to hash methods need to first learn continuous representations and then generate binary hash codes in a separated binarization step, which suffer from substantial loss of retrieval quality. This work presents HashNet, a novel deep architecture for deep learning to hash by continuation method with convergence guarantees, which learns exactly binary hash codes from imbalanced similarity data. The key idea is to attack the ill-posed gradient problem in optimizing deep networks with non-smooth binary activations by continuation method, in which we begin from learning an easier network with smoothed activation function and let it evolve during the training, until it eventually goes back to being the original, difficult to optimize, deep network with the sign activation function. Comprehensive empirical evidence shows that HashNet can generate exactly binary hash codes and yield state-of-the-art multimedia retrieval performance on standard benchmarks.
1. Introduction
Learning to hash supports efficient multimedia retrieval by mapping data to compact binary codes, while deep methods jointly learn representations and hash functions. Truly end-to-end training remains difficult because sign activations create ill-posed gradients and sparse similarity labels create data imbalance.
- Hashing transforms high-dimensional media data into compact binary codes so similar items receive similar codes for efficient approximate nearest-neighbor retrieval.
- Deep learning to hash jointly learns feature representations and hash coding with neural networks, enabling nonlinear hash functions and strong benchmark performance.
- Sign activation is required for exactly binary codes, but its zero gradient for nonzero inputs makes standard back-propagation infeasible.
- Sparse similarity labels create imbalance because similar pairs are much fewer than dissimilar pairs, making similarity-preserving learning ineffective.
- HashNet addresses both challenges with continuation-based optimization and a weighted pairwise cross-entropy loss in an end-to-end framework.
2. Related Work
Prior hashing work includes unsupervised and supervised approaches, while deep methods improve representation learning and hash coding. Existing deep hashing methods nevertheless learn continuous codes and binarize them afterward, leaving a gap from the exact binary hashing objective.
- Unsupervised hashing learns binary codes from unlabeled data, whereas supervised hashing uses semantic labels or relevance information to mitigate the semantic gap.
- Deep hashing methods progressively moved from two-stage training toward joint feature learning and hash coding, with cross-entropy and quantization losses improving retrieval performance.
- Existing deep hashing methods learn continuous codes and use a separate binarization step to produce binary codes.
- Continuous relaxation optimizes objectives that deviate from the binary hashing objective because quantization error cannot be reduced to zero.
- Direct optimization of binary codes requires sign activation inside the network, which introduces the ill-posed gradient problem.
3. HashNet
HashNet combines deep representation learning, sign-based binary encoding, weighted similarity learning, and continuation optimization. It progressively replaces smooth activations with sign activation while preserving loss descent and convergence-stage guarantees.
- Architecture: HashNet maps pairwise images through a CNN and fully connected hash layer to K-dimensional representations, then uses sign activation to produce K-bit binary codes.
- Learning by continuation: Continuation starts with tanh(βz), increases β across training stages, and converges toward the original sign-activation problem as β approaches infinity.
- Weighted similarity learning: Weighted maximum likelihood assigns pair-specific weights according to the importance of misclassification, addressing imbalance between similar and dissimilar training pairs.
- Weighted similarity learning: The pairwise logistic model uses hash-code inner products to assign higher similarity probability to pairs with smaller Hamming distance.
- Binary encoding: HashNet directly uses h = sgn(z), converting continuous hash-layer representations into values of +1 or −1.
- Convergence analysis: Theoretical analysis states that HashNet loss remains unchanged across stages and decreases during stochastic-gradient optimization within each stage.
4. Experiments
HashNet is evaluated on three benchmark image-retrieval datasets using multiple retrieval metrics and comparisons. It achieves stronger retrieval quality, handles imbalanced similarity data, and produces more discriminative exactly binary codes.
- Experimental setup: Experiments evaluate HashNet on ImageNet, NUS-WIDE, and MS COCO using five standard retrieval metrics and ten comparison methods.The metrics include MAP, precision-recall, P@H=2, P@N, and code histograms without binarization.
- MAP results: HashNet improves average MAP over DHN by 14.6%, 3.7%, and 2.9% on ImageNet, NUS-WIDE, and MS COCO, respectively.It also exceeds the best shallow methods using deep features by 15.7%, 15.5%, and 9.1% on the same datasets.
- Retrieval results: HashNet achieves the highest P@H=2 on all three datasets, with its 32-bit result exceeding DHN at every tested code length.P@H=2 matters for efficient Hamming ranking because each query requires O(1) time.
- Retrieval results: HashNet yields higher precision at low recall and for small top-result sets, supporting precision-first retrieval in practical systems.Qualitative top-10 examples likewise show more relevant and user-desired results.
- Protocol analysis: HashNet substantially outperforms DHN under the ImageNet zero-shot retrieval protocol.This indicates strong performance under both supervised and zero-shot retrieval protocols reported in the experiments.
- Code visualization: HashNet codes show clear category separation in t-SNE visualizations, whereas DHN codes lack comparable discriminative structure.The visualization suggests more discriminative hash codes for similarity retrieval.
- Imbalance analysis: Weighted maximum likelihood improves average MAP over HashNet-W by 12.4%, 2.8%, and 0.1% on ImageNet, NUS-WIDE, and MS COCO, respectively.The larger ImageNet gain reflects its stronger dissimilar-to-similar pair imbalance, whereas MS COCO is balanced.
- Continuation analysis: Continuation improves average MAP over HashNet-sgn by 8.1%, 1.4%, and 3.0% on ImageNet, NUS-WIDE, and MS COCO, respectively.Unlike surrogate-activation methods requiring post-step binarization, HashNet uses continuation with sign activation to learn lossless binary codes.
5. Conclusion
HashNet addresses deep hashing for imbalanced similarity data using continuation, learning exactly binary codes and achieving state-of-the-art multimedia retrieval performance on standard benchmarks.
- HashNet applies continuation to imbalanced similarity data in deep convolutional networks.
- HashNet optimizes a novel weighted pairwise cross-entropy loss and uses multi-stage pre-training.
- HashNet generates exactly binary hash codes and achieves state-of-the-art multimedia retrieval performance on standard benchmarks.
7. Supplemental Material: HashNet: Deep Learning to Hash by Continuation
The supplemental analysis establishes that continuation preserves HashNet’s binary-code loss across stages and decreases it during within-stage optimization, supporting convergence to sign-activated binary hashing.
- Convergence Analysis: The continuation procedure decreases HashNet’s loss at every stage and iteration until convergence to sign activation with exactly binary codes.
- Convergence Analysis: HashNet’s continuous activation uses tanh(β_t z) and becomes binary as β_t approaches infinity, while h_i = sgn(g_i).
- Convergence Analysis: Continuation changes tanh bandwidth across stages while preserving the sign hash code, so HashNet’s binary-code loss remains unchanged.
- Convergence Analysis: The supplemental derivative cases show gradient directions for paired bits under dissimilar and similar labels.
- Convergence Analysis: SGD optimization within each continuation stage decreases the HashNet loss.