Source-linked AI summary
Shared Predictive Cross-Modal Deep Quantization
Erkun Yang, Cheng Deng, Chao Li, Wei Liu, Jie Li, Dacheng Tao
TL;DR
Cross-modal search needs to retrieve across heterogeneous modalities while preserving their similarities, but common-space approaches face storage and query-efficiency challenges. SPDQ jointly learns shared and private subspaces and label-aligned compact codes; on two benchmark datasets, it significantly outperforms state-of-the-art methods in search accuracy and efficiency.
Problem
Cross-modal retrieval must preserve intra- and inter-modal similarities across heterogeneous feature spaces, while common-space methods face storage and query-efficiency challenges.
Source: §I · p. 1Method
SPDQ jointly learns shared and modality-private representations and quantizes the shared representations with label alignment to produce compact codes.
Source: §I · p. 2Results
On two popular cross-modal datasets, SPDQ significantly outperforms state-of-the-art methods in search accuracy and search efficiency.
Source: §I · p. 2Takeaways & Limitations
The shared/private design aims to retain modality-specific properties while representing information shared across modalities.
Source: §I · p. 2
Abstract
from arXiv · showhide
With explosive growth of data volume and ever-increasing diversity of data modalities, cross-modal similarity search, which conducts nearest neighbor search across different modalities, has been attracting increasing interest. This paper presents a deep compact code learning solution for efficient cross-modal similarity search. Many recent studies have proven that quantization-based approaches perform generally better than hashing-based approaches on single-modal similarity search. In this paper, we propose a deep quantization approach, which is among the early attempts of leveraging deep neural networks into quantization-based cross-modal similarity search. Our approach, dubbed shared predictive deep quantization (SPDQ), explicitly formulates a shared subspace across different modalities and two private subspaces for individual modalities, and representations in the shared subspace and the private subspaces are learned simultaneously by embedding them to a reproducing kernel Hilbert space, where the mean embedding of different modality distributions can be explicitly compared. In addition, in the shared subspace, a quantizer is learned to produce the semantics preserving compact codes with the help of label alignment. Thanks to this novel network architecture in cooperation with supervised quantization training, SPDQ can preserve intramodal and intermodal similarities as much as possible and greatly reduce quantization error. Experiments on two popular benchmarks corroborate that our approach outperforms state-of-the-art methods.
I. INTRODUCTION
Cross-modal search must bridge heterogeneous feature spaces while preserving both within- and between-modality similarity, but common-space methods face storage and query-efficiency challenges. SPDQ jointly learns shared and modality-private representations and uses label alignment to produce compact codes that preserve semantic similarities.
I. INTRODUCTION
- Cross-modal retrieval must preserve intra- and inter-modal similarities across heterogeneous feature spaces, while common-space approaches face storage and query-efficiency limits and earlier deep methods isolate modalities.Compact coding is presented as a way to address the efficiency and storage challenges.Source: §I · pp. 1–2
- SPDQ jointly learns shared and modality-private subspaces, then applies label alignment to compact codes to reduce intra-class diversity and preserve intra- and inter-modal semantic similarities.The shared and private representations are learned together in an end-to-end architecture.Source: §I · p. 2
A. Cross-modal Hashing
- Cross-modal hashing maps modalities into a common Hamming space; approaches use similarity alignment, semantic affinities, intra- and inter-modal losses, or spectral methods.The related-work discussion also groups compact coding into hashing, quantization, and deep-learning approaches.Source: A. Cross-modal Hashing · p. 2, §II · p. 2
B. Cross-modal Quantization
Cross-modal quantization approximates data with learned quantizers, while deep methods combine representation learning and compact-code learning. SPDQ distinguishes itself by jointly modeling shared and private subspaces, with label alignment and quantization in the shared representation.
B. Cross-modal Quantization
- Product quantization separately quantizes subvectors, additive quantization sums codewords, and composite quantization adds inner-product constraints; cross-modal methods include QCH, CCQ, and collaborative quantization.These methods use modality alignment or shared representations to support compact cross-modal search.Source: B. Cross-modal Quantization · pp. 2–3
C. Deep Learning Based Cross-modal Methods
- Earlier deep methods use joint hash learning (DCMH [21]), CNN/RNN-generated codes (DVSH), or learned deep quantizers (CDQ); SPDQ adds explicit shared and private subspaces with label alignment.The paper describes SPDQ as the first attempt to jointly model a private subspace for each modality and a shared subspace between modalities.Source: C. Deep Learning Based Cross-modal… · p. 3
III. FORMULATION
- For paired image-text data with labels, CNNs learn shared and modality-private subspaces; shared representations use label alignment and additive quantization to obtain compact codes.The database formulation specifies image and text data points with corresponding class labels.Source: §III · p. 3
A. Shared Predictive Representation Learning
SPDQ separates cross-modal shared content from modality-specific content, using MK-MMD to shape both spaces and a classification loss to keep representations discriminative.
A. Shared Predictive Representation Learning
- Modeling private components alongside shared components can enhance shared-information capture because private and shared components are coupled in the original representations.The paper argues that explicitly modeling both components can support more accurate search.Source: A. Shared Predictive Representation… · p. 3
- The multi-kernel distance can accommodate changing feature distributions during learning, with different kernel components contributing at different times.A larger distance is obtained when the distributions are not similar.Source: A. Shared Predictive Representation… · p. 4
- MK-MMD compares modality distributions in an RKHS, encouraging same-class cross-modal representations to be similar in the shared space and modality-specific representations to be dissimilar in private spaces.The method applies MK-MMD as the distance function for learning representations in both spaces.Source: A. Shared Predictive Representation… · pp. 3–4
- Concatenating shared and private representations into complete representations supports classification training intended to avoid trivial solutions and improve discriminative ability.The classification loss is sigmoid cross-entropy.Source: A. Shared Predictive Representation… · p. 4
B. Quantization with Label Alignment
Label information identifies category-specific structures in the shared space, and additive quantization approximates those label-space representations with compact codes.
B. Quantization with Label Alignment
- The method assumes same-category multimodal data share structures correlated with their labels, which label information is used to identify and preserve.Representations sharing labels map to the same category-specific surrogate, reducing intra-class diversity.Source: B. Quantization with Label Alignment · pp. 4–5
- Transformation matrices align image and text shared-space representations in the label space, where the auxiliary matrix Z represents category-specific structures.The class-level surrogate ZL stands in for the transformed image and text representations.Source: B. Quantization with Label Alignment · p. 5
- Additive quantization approximates ZL as a sum of selected elements from M dictionaries, with each indicator selecting one of K elements per dictionary.The quantization objective combines the alignment and approximation terms.Source: B. Quantization with Label Alignment · p. 5
C. Overall Objective Function
The overall objective joins shared predictive representation learning with quantization, allowing the two parts to be optimized jointly.
C. Overall Objective Function
- The overall objective combines shared predictive representation learning with quantization into compact codes using label alignment.The objective is subject to the stated projection constraints.Source: C. Overall Objective Function · p. 5
- The weight λ balances the representation-learning and quantization parts of the overall objective.Source: C. Overall Objective Function · p. 5
- Jointly minimizing the objective is intended to learn representations better suited to quantization and generate more effective compact codes.Source: C. Overall Objective Function · p. 5
D. Search Process
SPDQ searches across modalities using asymmetric quantizer distance (AQD) and trains its network and quantization components through alternating updates. Its optimization uses closed-form updates where available and ICM for compact codes.
D. Search Process
- The image query is mapped to a common representation and label-space representation, then AQD measures its distance to text database points.The search uses inner-product distance.Source: D. Search Process · p. 5
- A lookup table precomputes inner products for dictionaries and code values; each AQD calculation uses M lookups and M additions, slightly more than Hamming distance.Source: D. Search Process · p. 5
IV. OPTIMIZATION
- Optimization alternates between updating network parameters with quantization parameters fixed and updating quantization parameters with the networks fixed.The network update adopts an unbiased estimate of MK-MMD.Source: §IV · p. 5
- The dictionary update has an analytic solution, while compact-code optimization is approximately solved by ICM, which is guaranteed to converge.The dictionary subproblem is an unconstrained quadratic problem; ICM checks dictionary elements while other code entries are fixed.Source: §IV · p. 6
A. Setup
Experiments use image–text pairs from FLICKR25K and NUS-WIDE, with dataset-specific filtering and train, validation, and test splits. Retrieval is evaluated by MAP and top-N precision against supervised cross-modal baselines.
A. Setup
- FLICKR25K experiments retain 20,015 labeled image–tag pairs, while NUS-WIDE uses 186,577 pairs from its 10 most popular concepts.FLICKR25K keeps tags appearing at least 20 times; NUS-WIDE is drawn from 269,648 image-tag pairs annotated across 81 concepts.Source: A. Setup · p. 8
- For FLICKR25K, 2,000 instances are held out for testing and 10,000 for training; NUS-WIDE uses 1,866 test instances and 10,000 training instances.Source: A. Setup · p. 8
- MAP and top-N precision evaluate Hamming-ranked retrieval, with relevance defined by sharing at least one label and MAP using R = 50.Source: A. Setup · p. 8
- Six supervised baselines are compared, including deep hashing DCMH and deep quantization CDQ; SPDQ uses AlexNet for images and a three-layer MLP for text.Experiments average five runs; dictionary size is K = 256.Source: A. Setup · pp. 8–9
B. Results and Discussions
SPDQ outperforms the compared shallow and deep baselines in MAP with CNN features and in top-N precision across the reported feature settings. Deep methods also outperform shallow baselines using hand-crafted features.
B. Results and Discussions
- With hand-crafted features, deep methods DCMH, CDQ, and SPDQ outperform all shallow baselines by a large margin.The authors suggest superior deep features and joint feature/code learning as possible explanations.Source: B. Results and Discussions · p. 9
- Most shallow methods gain MAP across code lengths when using CNN features rather than hand-crafted features.Source: B. Results and Discussions · p. 9
- SPDQ outperforms all shallow CNN-feature baselines across retrieval tasks and also exceeds deep hashing DCMH and deep quantization CDQ.Source: B. Results and Discussions · p. 9
- SPDQ leads the reported top-N precision curves with both hand-crafted and CNN features, particularly when relatively few results are returned.The text reports consistency with the MAP evaluation and notes that the curves cover FLICKR25K and NUS-WIDE.Source: B. Results and Discussions · p. 9
C. Effect of Training Size
The training-size analysis varies the number of training examples at a fixed 32-bit code length and measures MAP on FLICKR25K and NUS-WIDE. SPDQ generally performs better with larger training sets, with a clearer increase on NUS-WIDE.
C. Effect of Training Size
- The analysis fixes code length at 32 bits, varies training size from 1,000 to 15,000, and uses the 10,000-example MAP as baseline.Source: C. Effect of Training Size · p. 9
- The supplied parameter plots show MAP versus α and MAP versus β.Source: Figure 8 · p. 10, Figure 9 · p. 10
- SPDQ usually achieves better retrieval performance with larger training sets, and the increase is particularly apparent on NUS-WIDE.The authors suggest NUS-WIDE's larger size and diversity may require more training data to capture dataset relationships.Source: C. Effect of Training Size · pp. 9–10
D. Parameter Sensitivity Analysis
The conclusion reports that SPDQ surpasses existing methods on two benchmark datasets, while parameter analysis shows that MAP depends on choosing γ, α, and β appropriately.
D. Parameter Sensitivity Analysis
- MAP rises as α increases from 0.1 to 1, but excessive α lets classification loss dominate quantization learning; β improves performance up to 1, then larger values decrease it.The authors select α = 1 and β = 1; β trades off quantization error against approximate error in label alignment.Source: D. Parameter Sensitivity Analysis · p. 10
- MAP is relatively stable for γ from 10^-5 to 5 × 10^-3 and decreases apparently above 0.005 on both datasets.The analysis fixes the code length at 32 bits; Fig. 10 reports MAP versus γ.Source: D. Parameter Sensitivity Analysis · p. 10, Figure 10 · p. 10
VI. CONCLUSIONS
- SPDQ surpasses existing methods on two benchmark multi-modal datasets.The conclusion attributes this approach to shared and private subspaces and label alignment in quantization training.Source: §VI · p. 10