Source-linked AI summary
Central Similarity Quantization for Efficient Image and Video Retrieval
Li Yuan, Tao Wang, Xiaopeng Zhang, Francis EH Tay, Zequn Jie, Wei Liu, Jiashi Feng
TL;DR
Existing hashing methods rely on local pairwise or triplet relationships, creating efficiency and coverage limitations. The paper introduces globally oriented central similarity, constructs separated hash centers, and trains Central Similarity Quantization for image and video hashing. CSQ reports improved retrieval performance and faster training, while its precomputed, data-independent centers leave distinctness among individual dissimilar pairs unmodeled.
Problem
Pairwise and triplet hashing captures similarity locally, making exhaustive learning inefficient and providing insufficient coverage of the data distribution.
Method
CSQ optimizes distances between hash codes and corresponding, sufficiently separated hash centers using central similarity, with centers generated by Hadamard matrices or Bernoulli sampling.
Results
3%-20% in mAP and 3 to 5.5 × faster training speed are reported over the latest methods across image and video retrieval tasks.
Takeaways & Limitations
CSQ generates cohesive codes for similar data and dispersed codes for dissimilar data, with state-of-the-art performance reported for image and video retrieval.
Abstract
from arXiv · showhide
Existing data-dependent hashing methods usually learn hash functions from pairwise or triplet data relationships, which only capture the data similarity locally, and often suffer from low learning efficiency and low collision rate. In this work, we propose a new \emph{global} similarity metric, termed as \emph{central similarity}, with which the hash codes of similar data pairs are encouraged to approach a common center and those for dissimilar pairs to converge to different centers, to improve hash learning efficiency and retrieval accuracy. We principally formulate the computation of the proposed central similarity metric by introducing a new concept, i.e., \emph{hash center} that refers to a set of data points scattered in the Hamming space with a sufficient mutual distance between each other. We then provide an efficient method to construct well separated hash centers by leveraging the Hadamard matrix and Bernoulli distributions. Finally, we propose the Central Similarity Quantization (CSQ) that optimizes the central similarity between data points w.r.t.\ their hash centers instead of optimizing the local similarity. CSQ is generic and applicable to both image and video hashing scenarios. Extensive experiments on large-scale image and video retrieval tasks demonstrate that CSQ can generate cohesive hash codes for similar data pairs and dispersed hash codes for dissimilar pairs, achieving a noticeable boost in retrieval performance, i.e. 3\%-20\% in mAP over the previous state-of-the-arts. The code is at: \url{https://github.com/yuanli2333/Hadamard-Matrix-for-hashing}
1. Introduction
The paper identifies local pairwise/triplet similarity as inefficient and incomplete for large-scale hashing, then proposes central similarity and CSQ to learn globally structured hash codes for image and video retrieval.
- Motivation: Pairwise and triplet hashing captures relationships locally, limiting efficient coverage of the whole training dataset.Exhaustively learning all pairs or triplets is impractical at scale and may harm discriminability through incomplete distribution coverage.
- Central Similarity: Central similarity encourages similar data points to approach shared hash centers and dissimilar points to occupy different centers.A hash center is a sufficiently separated set of points in Hamming space.
- Central Similarity: O(nm) complexity enables central-similarity hashing to use global relationships across n data points and m centers.The method is described as remaining effective even under severe data imbalance.
- Hash Center Construction: Hash centers are constructed either with maximal mutual Hamming distance from a Hadamard matrix or by Bernoulli sampling.Both approaches are described as generating centers separated by sufficient Hamming distance.
- CSQ: CSQ combines CNN feature learning and a hash layer, training end-to-end to optimize central similarity for image and video hashing.Video hashing can use the same approach with an appropriate convolutional architecture.
- Results: 3%-20% in mAP and 3 to 5.5 × faster training speed are reported over the latest methods across image and video retrieval experiments.MNIST visualizations also show favorable intra-class compactness and inter-class separability compared with other state-of-the-art hashing methods.
2. Related Work
Prior image and video hashing methods commonly model similarity through pairwise or temporal relationships, whereas this work introduces central similarity to model relationships using hash centers for both modalities.
- Image Hashing: Deep image hashing methods commonly use 2D CNNs with pairwise or triplet similarity losses to learn hash codes.Examples include weighted pairwise, maximum-a-posterior, and triplet-ranking formulations.
- Central Similarity: CSQ proposes central similarity as a new metric for modeling relationships between similar and dissimilar pairs.The stated goal is to improve the discriminability of generated hash codes.
- Video Hashing: Video hashing methods incorporate temporal information through disaggregation, LSTM modeling, fully connected layers, or frame pooling.These methods differ from CSQ's generic formulation for image and video hashing.
- Relation to Center Loss: Unlike feature-space center loss, CSQ defines hash centers over hash codes and uses them to structure retrieval representations.The paper distinguishes its hash centers from category-derived feature centers that may be unstable with intra-class variation.
3. Method
CSQ replaces local pairwise or triplet similarity modeling with central similarity relative to well-separated hash centers. It generates centers systematically, associates them with single- or multi-label data, and optimizes hash codes toward those centers with quantization refinement.
- Central similarity: Hash centers are sufficiently separated binary points in the K-dimensional Hamming space, and central similarity encourages similar data toward shared centers and dissimilar data toward different centers.The formulation uses semantic hash centers associated with training samples to preserve global similarity information.
- Hash-center generation: Hadamard matrices generate mutually separated centers when K is a power of 2, while Bernoulli(0.5) sampling yields pairwise Hamming distance K/2 in expectation.Hadamard rows provide distance K/2 after converting -1 entries to 0; Bernoulli sampling supplies the alternative construction.
- Hash-center generation: The Hadamard construction is constrained when m exceeds 2K or K is not a power of 2, so random Bernoulli sampling is used in those cases.The first approach can combine two Hadamard matrices only when K < m <= 2K and K = 2^n.
- Semantic hash centers: For single-label data, one center is assigned per category; for multi-label data, category centers are combined by bitwise voting to form transitive semantic centers.Ties in bitwise voting are resolved by sampling from Bern(0.5).
- Central Similarity Quantization: CSQ uses CNN feature layers and a hash layer trained end-to-end against semantic hash centers, with quantization loss refining codes toward binary centers.The central similarity objective is derived through MAP estimation, while the quantization term addresses incomplete convergence to binary centers.
4. Experiments
Experiments evaluate CSQ across image and video retrieval benchmarks, backbones, metrics, training time, retrieval visualizations, and hash-center behavior. CSQ consistently improves retrieval performance, trains faster, and produces compact codes for similar data while separating dissimilar data.
- Experimental setup: CSQ is evaluated on five benchmark datasets spanning ImageNet, NUS WIDE, MS COCO, UCF101, and HMDB51.
- Image retrieval: Image retrieval experiments compare CSQ with nine classical or state-of-the-art hashing and quantization methods using mAP and related precision metrics.
- Image retrieval: At least 11.5%, 3.6%, and 3.1% mAP increases over HashNet and DCH are reported on ImageNet, MS COCO, and NUS WIDE, respectively.Some image-retrieval improvements reach 20%.
- Video retrieval: Video retrieval shows mAP increases of over 12.0% on UCF101 and 4.8% on HMDB51, with CSQ outperforming compared methods across three UCF101 metrics.
- Visualization and analysis: For multi-label MS COCO images, CSQ targets the centroid of multiple centers and returns images sharing more query labels than HashNet.
- Visualization and analysis: Heat maps show small diagonal Hamming distances to assigned centers and mostly large off-diagonal distances to other centers.This indicates generated codes collapse toward corresponding centers while dissimilar data spread across centers.
- Ablation study: The ablation study finds central similarity has positive effects, while adding pairwise similarity provides limited further improvement and removing central similarity substantially decreases mAP.
5. Conclusion and Future Work
The paper concludes that CSQ uses hash centers and central similarity to learn high-quality codes for image and video retrieval. It identifies data-independent center generation as effective while leaving better center learning for future work.
- CSQ optimizes Hamming distance between hash codes and corresponding hash centers to learn high-quality codes.
- Experiments validate state-of-the-art performance for both image and video retrieval.
- Hash centers generated independently of data features are reported effective, while learning better hash centers remains future work.