PaperlayerView exact revision on arXiv

Source-linked AI summary

MTFH: A Matrix Tri-Factorization Hashing Framework for Efficient Cross-Modal Retrieval

Xin Liu, Zhikai Hu, Haibin Ling, Yiu-ming Cheung

arXiv:1805.01963v2cs.CV

TL;DR

Cross-modal hashing commonly uses unified or equal-length codes despite heterogeneous modalities, differing feature dimensions, and potentially unpaired data. MTFH jointly learns modality-specific codes, semantic correlation matrices, and discrete mappings for paired or unpaired data with equal or varying lengths. Experiments report competitive or mostly improved retrieval performance, while the paper notes that highly different code lengths can degrade discriminative power.

  • Problem

    Unified or equal-length codes do not fully address heterogeneous feature dimensions, non-one-to-one correspondence, and the need to support paired or unpaired multi-modal data.

  • Method

    MTFH jointly learns modality-specific hash codes, two semantic correlation matrices, and discrete length-mapping factors for equal or varying hash lengths.

  • Results

    MTFH generalizes across paired or unpaired data and equal or varying hash lengths, with comparable or mostly improved retrieval performance across benchmark tasks.

  • Takeaways & Limitations

    Varying hash lengths can support heterogeneous representations and reduce storage while maintaining similar retrieval performance in reported settings.

  • Takeaways & Limitations

    Unequal hash lengths that differ substantially can degrade mapping-code discriminative power, making appropriate length selection necessary.

Abstract

from arXiv · show

Hashing has recently sparked a great revolution in cross-modal retrieval because of its low storage cost and high query speed. Recent cross-modal hashing methods often learn unified or equal-length hash codes to represent the multi-modal data and make them intuitively comparable. However, such unified or equal-length hash representations could inherently sacrifice their representation scalability because the data from different modalities may not have one-to-one correspondence and could be encoded more efficiently by different hash codes of unequal lengths. To mitigate these problems, this paper exploits a related and relatively unexplored problem: encode the heterogeneous data with varying hash lengths and generalize the cross-modal retrieval in various challenging scenarios. To this end, a generalized and flexible cross-modal hashing framework, termed Matrix Tri-Factorization Hashing (MTFH), is proposed to work seamlessly in various settings including paired or unpaired multi-modal data, and equal or varying hash length encoding scenarios. More specifically, MTFH exploits an efficient objective function to flexibly learn the modality-specific hash codes with different length settings, while synchronously learning two semantic correlation matrices to semantically correlate the different hash representations for heterogeneous data comparable. As a result, the derived hash codes are more semantically meaningful for various challenging cross-modal retrieval tasks. Extensive experiments evaluated on public benchmark datasets highlight the superiority of MTFH under various retrieval scenarios and show its competitive performance with the state-of-the-arts.

1 INTRODUCTION

Cross-modal hashing must handle heterogeneous feature spaces, imperfect correspondence, and code-length choices that affect representation quality. MTFH addresses these constraints with a flexible framework supporting paired or unpaired data and equal or varying hash lengths.

  • Motivation: Cross-modal retrieval searches one modality for semantically relevant samples in another, but heterogeneous feature spaces make similarity measurement difficult.Examples include retrieving images for textual queries or text for visual queries.
  • Motivation: Hash length trades off discriminative power and redundancy, so forcing one common length may differ from the optimal lengths for individual modalities.Longer codes can reduce false collisions and generally improve precision, but code-length selection remains important.
  • Motivation: Unified or equal-length codes can limit representation scalability because modalities may have different feature dimensions and non-one-to-one semantic correspondence.The paper motivates varying hash lengths as a better fit for heterogeneous representations.
  • Proposed framework: MTFH is a generalized framework for paired or unpaired multi-modal data with equal or varying hash-length encoding.It constructs a semantic-label affinity matrix and jointly learns modality-specific hash codes.
  • Proposed framework: MTFH learns varying-length codes for heterogeneous data and uses an efficient discrete optimization algorithm without relaxation to reduce quantization error.The paper presents this as its first attempt to learn different-length hash codes for cross-modal comparison.
  • Evaluation: Experiments on public benchmarks report comparable or mostly improved retrieval performance over existing methods across various cross-modal retrieval tasks.The paper evaluates MTFH under multiple retrieval scenarios.

2 RELATED WORKS

Related work includes common-subspace, conventional hashing, supervised and unsupervised cross-modal hashing, and deep hashing approaches. These methods address heterogeneity and semantic gaps but commonly retain unified codes or paired-data requirements.

  • Common-space methods: Cross-modal retrieval methods often project heterogeneous data into a common subspace to minimize differences between modalities.Canonical correlation analysis is cited as an approach that maximizes correlations between projected vectors.
  • Hashing methods: Hashing is attractive for cross-modal retrieval because it offers low storage cost and fast query speed through compact binary codes.Earlier single-modal hashing methods provide theoretical foundations for cross-modal hashing.
  • Supervision: Unsupervised cross-modal hashing can remain constrained by the semantic gap between high-level semantic descriptions and low-level feature descriptors.The cited discussion connects this constraint with degraded retrieval performance.
  • Supervision: Supervised methods use semantic labels or relevance feedback to mitigate the semantic gap and produce compact codes for retrieval.Examples include semantic correlation maximization and semantic preserving hashing.
  • Deep methods: Deep cross-modal hashing integrates feature and code learning but is constrained by computational complexity, optimization cost, unified codes, and reliance on paired data.The discussion also notes a mismatch between Hamming distance and metric distance on high-level representations.

3 MATRIX TRI-FACTORIZATION HASHING

MTFH extends cross-modal hashing beyond unified, equal-length codes by learning modality-specific representations with flexible lengths and semantic correlations. Its tri-factorization framework supports paired or unpaired data, while discrete optimization and learned hash functions enable retrieval for new samples.

  • MTFH addresses the lack of varying-length cross-modal hashing for heterogeneous data and supports paired or unpaired collections.
  • A basic factorization produces equal-length codes, whereas MTFH introduces q1- and q2-length representations to accommodate different modality encodings.
  • The framework factorizes semantic affinity into modality-specific hash matrices and correlation matrices that align representations with unequal lengths.H1 maps Y's code length from q2 to q1, while H2 supports the reverse mapping from q1 to q2.
  • The optimization is non-convex and discrete, so MTFH separates binary constraints with auxiliary variables and uses discrete coordinate descent rather than continuous relaxation.The paper notes that relaxation can accumulate quantization error as code length increases.
  • MTFH learns modality-specific hash functions from input features to binary codes and transforms independently learned codes through H1 and H2 for cross-modal retrieval.

4 EXPERIMENTS

The experiments evaluate MTFH on public multi-modal benchmarks using mean Average Precision across cross-modal retrieval queries. The evaluation spans datasets with different sizes, feature dimensions, semantic labels, and sampling protocols.

  • 4.1 Datasets and Evaluation Protocol: MTFH is evaluated quantitatively on three public benchmarks: Wiki, MIRFlickr, and NUS-WIDE.
  • 4.1 Datasets and Evaluation Protocol: Wiki contains 2,866 image-text pairs across 10 categories, with 2,173 training instances and 693 testing instances.Images use 128-dimensional SIFT features, while text uses 10-dimensional LDA features.
  • 4.1 Datasets and Evaluation Protocol: MIRFlickr comprises 25,000 image-text pairs with 24 semantic labels and uses 150-dimensional image features and 500-dimensional text features.
  • 4.1 Datasets and Evaluation Protocol: NUS-WIDE includes 269,548 image-text pairs and 81 concepts, with NUS-WIDE-100k formed from 100,000 labeled pairs for scalable training and querying.The 100,000-pair setup uses 5% of pairs as the query set and the remainder as training data.
  • 4.1 Datasets and Evaluation Protocol: Performance is measured by mean Average Precision over all queries, where relevant neighbors share at least one semantic label with the query.

4.2 Baseline Methods

MTFH is evaluated against eight established cross-modal hashing baselines using author-provided implementations and reported parameter settings. Across datasets, it is competitive on Wiki and strongest on larger multi-label benchmarks, while top-ranked retrieval precision remains high.

  • Baseline Methods: The study compares MTFH with eight well-known cross-modal hashing methods treated as current state-of-the-art baselines.Implementations use source code and parameter settings provided by the respective authors.
  • Experimental Setup: For fair comparison, equal-length codes set q1=q2 and evaluate 16, 32, 64, and 128 bits using random and k-means sampling.mAP scores are recorded across four benchmark datasets.
  • Results: MTFH is competitive on Wiki and delivers the best retrieval performance on the larger datasets.DCH and SRLCH are particularly competitive on selected Wiki tasks, but their performance often degrades on larger datasets.
  • Results: On larger multi-label datasets, MTFH’s modality-specific codes are more semantically meaningful than unified representations for cross-modal retrieval.The paper attributes the advantage to stronger characterization of heterogeneous samples and improved latent-structure modeling.
  • Results: MTFH produces the highest topK-precision curves as K changes in representative MIRFlickr and NUS-WIDE-100k evaluations.It is comparable to DCH on MIRFlickr mAP@50 and outperforms state-of-the-art baselines on NUS-WIDE-100k mAP@50.

4.4 Results of Unequal Hash Length Encoding

MTFH supports unequal hash lengths across modalities, and experiments show that unequal settings can match or exceed equal-length encoding. The best setting depends on the dataset, task, and modality-specific code lengths rather than simply increasing every code.

  • Framework: MTFH is presented as the first framework for generating different-length hash codes for multi-modal representation.It evaluates unequal image-text settings such as I-16 and T-32.
  • Curve Evaluations: Precision-recall curves compare approaches on different datasets at representative code lengths of 32, 64, and 128 bits.These curves provide performance comparisons across selected code lengths rather than a single hash-length setting.
  • Curve Evaluations: TopK-precision curves are reported on MIRFlickr and NUS-WIDE-100k, while additional retrieval results vary one modality’s code length while fixing the other.The latter evaluation is summarized in Fig. 7.
  • Unequal Hash Length Results: Larger codes do not always improve retrieval, and I-80&T-100 outperforms I-100&T-100 on MIRFlickr.This shows that shortening one modality’s code can preserve or improve performance.
  • Recall Evaluation: On Wiki, recall is evaluated with image codes fixed at 128 bits while text-code lengths vary from 16 to 128 bits.Equal-length comparisons include I-16&T-16, I-32&T-32, and I-64&T-64.

4.5 Results of the Unpaired Scenario

MTFH is evaluated on unpaired multi-modal collections with single or multiple labels and without one-to-one correspondence. It outperforms or remains more stable than competing methods across these retrieval settings.

  • Unpaired data settings: Unpaired evaluation covers single-label and multi-label collections without one-to-one correspondence between modalities.Wiki is used for single-label unpaired data, while MIRFlickr is used for multi-label unpaired data.
  • Retrieval performance: MTFH significantly outperforms baseline methods on unpaired multi-modal data collections.CCA, IMH, CMFH, and GSePH obtain relatively lower or degraded mAP scores in these settings.
  • Retrieval performance: MTFH achieves I→T mAP values higher than 0.73 on the evaluated unpaired tasks, whereas GSePH remains below 0.69 on MIRFlickr.The comparison is reported for unpaired retrieval on MIRFlickr.
  • Retrieval performance: MTFH produces more stable MIRFlickr retrieval performance than DCH across different retrieval tasks.The paper reports that MTFH mAP values are always higher than DCH results on MIRFlickr.

4.6 Results of Single-modal Retrieval

MTFH supports single-modal retrieval using modality-specific hash codes, including unequal code lengths. Its equal-length codes outperform representative alternatives, while I-128&T-64 yields the best reported I→I result on NUS-WIDE-100k.

  • Method and representation: MTFH learns different hash codes for heterogeneous modalities that can support single-modal retrieval.This contrasts with unified hash codes, which produce the same representation for heterogeneous data points.
  • Equal-length encoding: MTFH equal-length hash codes outperform ITQ, SGH, FSDH, and FSH on representative single-modal retrieval datasets.The comparison is reported using mAP results in Table 5.
  • Varying-length encoding: MTFH jointly learns modality-specific hash codes with different length settings for single-modal retrieval.The framework is described as supporting varying-length representations rather than requiring equal code lengths.
  • Varying-length encoding: I-128&T-64 produces the best I→I retrieval performance on NUS-WIDE-100k.The paper attributes this result to the greater semantic meaningfulness of the coupled code lengths for that task.

4.7 Results of CNN Visual Features

MTFH is tested with fine-tuned CNN visual features on Wiki, Pascal Sentence, and Pascal VOC 2007. It generally outperforms competing baselines across hash lengths, especially on Wiki and Pascal VOC 2007.

  • Experimental setting: MTFH is evaluated on Wiki, Pascal Sentence, and Pascal VOC 2007 using publicly shared fine-tuned CNN visual features.The evaluation targets cross-modal retrieval with CNN-based visual representations.
  • Cross-modal retrieval results: MTFH and DCH outperform CCA, T-V CCA, deep-SM, CMFH, SePH, and GSePH on the CNN-feature evaluation.The comparison is reported in Table 6 using cross-modal retrieval mAP results.
  • Cross-modal retrieval results: MTFH often improves retrieval performance across different hash length settings and significantly outperforms most baselines, especially on Wiki and Pascal VOC 2007.DCH is competitive on Pascal Sentence but degrades on Wiki and Pascal VOC 2007.
  • Cross-modal retrieval results: At 128 bits on Wiki, MTFH mAP exceeds 0.5 for I→T and 0.8 for T→I.The result is presented as evidence that the learned hash projection functions map CNN visual features into compact hash codes.

4.8 Effects of Discrete Optimization

The paper compares DCC with the proposed E-RCD for optimizing the same MTFH objective. E-RCD retains competitive retrieval performance while producing more stable results across repeated trials.

  • Optimization design: MTFH uses a discrete optimization algorithm to jointly learn modality-specific hash codes without relaxation.The motivation is to avoid quantization error that can accumulate as code length increases.
  • Optimization design: DCC updates hash bits cyclically and may yield unstable retrieval performance because it is an approximate discrete optimization solution.The paper also notes that DCC may fall into a local minimum during learning.
  • Repeated-trial evaluation: 0.3342 and 0.7284 are the average 128-bit mAP values from ten DCC trials for I→T and T→I on Wiki.The directly optimized MTFH objective still outperforms most listed state-of-the-art baselines.
  • Repeated-trial evaluation: E-RCD yields competitive performance with lower max-min differences and standard deviations than DCC across ten trials.The reported average mAP values do not change significantly for E-RCD across repeated runs.

4.9 Parameter Sensitivity Analysis

MTFH’s parameter study sets α=0.5 and λ=0.1 in most cases, while training-set experiments show that performance improves consistently with larger samples.

  • Parameter Sensitivity Analysis: α=0.5 balances hash-code learning across the two modalities.α controls the balance between learning the q1-bit code for X and the q2-bit code for Y.
  • Parameter Sensitivity Analysis: λ=0.1 is used in most cases because it is insensitive to the least-square optimization.
  • Parameter Sensitivity Analysis: Around 10k training samples for I→T and 30k for T→I make MTFH outperform SePH on NUS-WIDE-100k.MTFH requires a somewhat larger training set than SePH to produce promising results.
  • Parameter Sensitivity Analysis: MTFH’s mAP scores increase consistently as the training-set size grows.

4.10 Discussion and Analysis

MTFH supports flexible unequal-length hashing and can reduce storage while maintaining or improving retrieval performance, but suitable code lengths remain necessary and computational scaling requires care.

  • Discussion and Analysis: MTFH’s computational cost is concentrated in matrix multiplications that can be parallelized, while large databases may require smaller training sets for hash-function learning.The semantic correlation matrix adds little retrieval-time overhead because it is small.
  • Discussion and Analysis: 96n−320 bits are saved when I-32&T-128 matches I-128&T-128 performance, and MTFH improves over SePH km on NUS-WIDE-All while saving around 17M bits.The proposed representation stores q1+q2 bits instead of 2q1 bits for paired instances when q2<q1.
  • Discussion and Analysis: Under similar memory budgets, varying pairs including I-48&T-80 and I-32&T-96 can outperform equal-length I-64&T-64 baselines.On MIRFlickr I→T retrieval, I-48&T-80 achieves the largest mAP score among the cited settings.
  • Discussion and Analysis: Unequal code lengths can degrade retrieval when the bit allocation differs substantially, as shown by I-128&T-16 on Wiki I→T.Appropriate length selection is therefore necessary for heterogeneous representations.
  • Discussion and Analysis: MTFH jointly learns unequal-length modality-specific codes and semantic correlation matrices so heterogeneous representations remain comparable.This joint training avoids the weakened connections and direct-comparison problem of separately learned varying-length codes.
  • Discussion and Analysis: MTFH generalizes across paired or unpaired data and equal or varying hash-length scenarios, whereas several prior extensions rely on paired data or unified codes.

5 CONCLUSION

The conclusion presents MTFH as a generalized framework for cross-modal retrieval that jointly learns semantically correlated hash codes with flexible lengths and supports diverse data settings.

  • 5 CONCLUSION: MTFH handles paired or unpaired multi-modal data with equal or varying hash-length encodings.
  • 5 CONCLUSION: MTFH jointly learns modality-specific codes and two semantic correlation matrices to preserve cross-modal comparability.
  • 5 CONCLUSION: An efficient discrete optimization algorithm without relaxation produces hash codes intended to preserve multi-modal semantic structure.
  • 5 CONCLUSION: Experiments across retrieval tasks verify MTFH’s effectiveness, and the paper identifies it as the first attempt to learn comparable varying-length hash codes for heterogeneous data.
Loading 1805.01963v2…