PaperlayerView exact revision on arXiv

Source-linked AI summary

Triplet-Based Deep Hashing Network for Cross-Modal Retrieval

Cheng Deng, Zhaojia Chen, Xianglong Liu, Xinbo Gao, Dacheng Tao

arXiv:1904.02449v1cs.IR

TL;DR

Existing cross-modal hashing methods often ignore relative semantic similarity between heterogeneous data, limiting their ability to learn powerful hash codes. TDH uses triplet supervision with inter- and intra-modal losses plus graph regularization, and outperforms several state-of-the-art approaches on two datasets.

  • Problem

    Existing cross-modal hashing methods often overlook relative semantic relationships between heterogeneous data, despite their richer semantic information.

  • Method

    TDH integrates triplet labels, inter- and intra-modal loss functions, and graph regularization into an end-to-end cross-modal hashing framework.

  • Results

    TDH outperforms several state-of-the-art cross-modal hashing approaches on the MIRFlickr25k and NUS-WIDE datasets across different retrieval tasks.

  • Takeaways & Limitations

    Triplet supervision captures relative semantic correlations, while graph regularization preserves semantic similarities in Hamming space within the proposed method.

Abstract

from arXiv · show

Given the benefits of its low storage requirements and high retrieval efficiency, hashing has recently received increasing attention. In particular,cross-modal hashing has been widely and successfully used in multimedia similarity search applications. However, almost all existing methods employing cross-modal hashing cannot obtain powerful hash codes due to their ignoring the relative similarity between heterogeneous data that contains richer semantic information, leading to unsatisfactory retrieval performance. In this paper, we propose a triplet-based deep hashing (TDH) network for cross-modal retrieval. First, we utilize the triplet labels, which describes the relative relationships among three instances as supervision in order to capture more general semantic correlations between cross-modal instances. We then establish a loss function from the inter-modal view and the intra-modal view to boost the discriminative abilities of the hash codes. Finally, graph regularization is introduced into our proposed TDH method to preserve the original semantic similarity between hash codes in Hamming space. Experimental results show that our proposed method outperforms several state-of-the-art approaches on two popular cross-modal datasets.

I. INTRODUCTION

Cross-modal hashing addresses retrieval across heterogeneous modalities but existing approaches often overlook relative semantic similarity. TDH uses triplet supervision, inter- and intra-modal losses, and graph regularization to learn more discriminative hash codes.

  • Cross-modal retrieval must bridge the semantic gap between heterogeneous data modalities that remain semantically correlated.
  • Traditional subspace methods face high computational complexity and low search accuracy as multimedia data increase, motivating hashing-based retrieval.
  • Pairwise labels encode only binary similarity, whereas triplet labels represent relative ordering and capture higher-level semantic relationships.
  • The method extends triplet-label supervision from same-modality image retrieval to cross-modal retrieval involving different query and result modalities.
  • TDH integrates triplet sampling and similarity losses in an end-to-end cross-modal hashing framework to capture relative semantic relationships.
  • TDH combines inter-modal and intra-modal triplet likelihood losses with graph regularization to enhance hash-code discrimination and preserve semantic similarities in Hamming space.

III. TRIPLET-BASED DEEP HASHING NETWORK FOR CROSS-MODAL RETRIEVAL

TDH is formulated as an end-to-end cross-modal hashing framework using image and text modalities. It defines triplet relationships and seeks hash functions whose Hamming distances preserve those relative similarities.

  • TDH integrates feature learning and hash-code learning into a unified end-to-end framework for cross-modal retrieval.
  • The formulation uses image and text to explain a method that can be expanded to multiple modalities.
  • A triplet (q_m, p_m, n_m) specifies that query q_m is more similar to positive p_m than to negative n_m.
  • The learned binary hash functions map text and images into k-bit codes and should preserve triplet semantics through Hamming-distance ordering.

B. Deep Architecture

TDH uses separate deep networks for images and text, then models triplet relationships over their learned hash representations. Its objective combines inter-modal, intra-modal, and graph-regularization losses.

  • TDH applies one deep neural network to each modality, using CNN-based image processing and MLP-based textual processing.
  • The image network replaces CNN-F’s fc8 layer with a k-node fully connected hash layer that maps learned features directly into Hamming space.
  • Text is represented with bag-of-words and processed by an MLP whose final layer is replaced by a k-node hash layer.
  • C. The Triplet Loss: Triplet loss increases similarity between query-positive pairs while reducing similarity between query-negative pairs.
  • The objective contains inter-modal triplet, intra-modal triplet, and graph-regularization losses to preserve semantic similarity and improve hash-code discriminability.

D. Inter-Modal Triplet Loss

The inter-modal triplet loss trains hash codes across image and text modalities by enforcing higher similarity for query-positive pairs than for query-negative pairs. It is defined for both image-to-text and text-to-image retrieval.

  • The inter-modal triplet embedding loss captures heterogeneous correlations across different modalities for cross-modal retrieval.
  • TDH defines separate inter-modal triplet losses for Image-to-Text and Text-to-Image retrieval.
  • The Image-to-Text and Text-to-Image formulations use modality-specific hash representations for queries, positives, and negatives.
  • Optimizing the loss decreases Hamming distance to positive instances and increases it to negative instances across modalities.

E. Intra-Modal Triplet Loss

The intra-modal triplet loss preserves semantic information within each modality and strengthens hash-code discrimination for cross-modal retrieval.

  • Intra-modal triplet loss improves hash-code discrimination by preserving semantic information within each modality.
  • The image modality uses an intra-modal triplet loss based on relationships among image hash codes.
  • The textual modality applies a corresponding intra-modal triplet loss to preserve within-text semantic relationships.

F. Graph Regularization Loss

Graph regularization preserves similarity among unified hash codes in Hamming space, and it forms one component of TDH’s joint objective optimized iteratively.

  • Graph regularization enhances correlation among unified hash codes in Hamming space.
  • The method constructs the graph from a label similarity matrix, with same-label pairs assigned similarity 1 and other pairs 0.
  • The graph loss is combined with inter-modal and intra-modal triplet losses in the overall TDH objective.
  • Mini-batch SGD and alternating learning optimize the network parameters and binary codes until convergence or a preset iteration limit.

A. Updating B

TDH updates its binary code matrix within an alternating optimization procedure that also trains the modality-specific networks using mini-batches and backpropagation.

  • The binary code matrix B is updated according to the optimization rule derived from the expanded objective.
  • The identity matrix I is used in the derivation of the binary-code update.
  • Algorithm 1 initializes network parameters, mini-batch sizes Nx = Ny = 128, and iteration counts before alternating updates.
  • After updating B, the algorithm samples text and image mini-batches, computes network outputs, and updates wx and wy through backpropagation.
  • The wx gradient is decomposed into inter-modal, intra-modal, and regularization components during optimization.

C. Updating wy

The image-network parameter wy is trained with SGD and backpropagation, while the learned modality-specific functions support out-of-sample hash generation and cross-modal retrieval.

  • The wy gradient is divided into inter-modal, intra-modal, and regularization components for SGD training.
  • Sampling P instances produces 3 × P triplet-label data, increasing the effective training data used in each iteration.
  • A new text instance is converted into a hash code by forward propagating through the learned textual network.
  • A new image instance is likewise converted into a hash code through the learned image network.
  • These modality-specific hash functions enable retrieval when query and result data come from different modalities.
  • TDH can be extended from two modalities to multiple modalities by extending the network structures and objective terms.

F. Triplet sample

TDH samples triplets from cross-modal datasets and uses them to train and evaluate retrieval models under specified similarity and dataset-construction procedures.

  • Triplet sampling: Each iteration samples P×M1×M2 triplets by combining positive and negative instances around an anchor.Positive instances are similar to the anchor, while negative instances are dissimilar.
  • Datasets: MIRFlickr25k retains 20,015 image-text pairs after excluding images with fewer than 20 textual tags.Text is represented by 1386-dimensional bag-of-words vectors; shallow image features use 512-dimensional SIFT vectors.
  • Similarity labels: Instances are considered similar when they share at least one label and dissimilar when they share none.
  • Datasets: NUS-WIDE selects its 10 most frequent concepts, leaving 186,577 image-text pairs for experimentation.

B. Evaluation Protocol

The evaluation tests cross-modal retrieval in both query directions using hash-based and Hamming-ranking metrics, against eight established baselines under controlled dataset splits and training settings.

  • Tasks and metrics: Evaluation covers text-query-image and image-query-text tasks using precision-recall, topN-precision, and MAP criteria.Hash lookup supports the first criterion, while Hamming ranking supports the latter two.
  • Tasks and metrics: MAP averages query-level Average Precision values, with larger MAP indicating better retrieval performance.Average Precision weights precision at retrieved ranks where results are relevant.
  • Baselines: TDH is compared with eight methods: CMFH, SCM, LSSH, STMH, CVH, SePH, DCMH, and PRDH.DCMH and PRDH are deep methods; the remaining baselines are shallow.
  • Splits and settings: MIRFlickr25k uses 2,000 queries and a randomly sampled 5,000-instance training set from the retrieval set.
  • Splits and settings: NUS-WIDE uses 1,866 queries and likewise samples 5,000 retrieval instances for training.
  • Splits and settings: The experiments use mini-batches of 128, 500 outer-loop iterations, and validation-selected parameters γ=100, η=50, and β=1.α is set to half the hash-code length.

E. Performance Comparisons and Disccussions

Across two datasets, retrieval tasks, code lengths, and feature settings, TDH generally outperforms the compared baselines; component analyses identify complementary roles for its losses.

  • Performance comparisons: TDH is superior to all baselines across the reported retrieval tasks and code lengths on MIRFlickr25k and NUS-WIDE.Tables III and IV report comparisons using hand-crafted features for text-query-image and image-query-text tasks.
  • Performance comparisons: TDH also outperforms all baselines when shallow methods use CNN-F features.The comparisons cover both cross-modal query directions on the two datasets.
  • Performance comparisons: With 32-bit codes, TDH curves remain higher than other methods for precision-recall and topN-precision evaluations using hand-crafted features.
  • Loss analysis: The inter-modal triplet loss has the largest objective-function effect, followed by the intra-modal triplet loss and then graph regularization.
  • Loss analysis: The three loss components are described as indispensable because they capture cross-modal correlation, hash-code discrimination, and semantic similarity preservation.
  • Parameter sensitivity: Larger α values improve performance in the reported sensitivity experiments, and α is set to half the hash-code length.The experiments use 16 bits on MIRFlickr25k and 32 bits on NUS-WIDE.
  • Conclusion: The conclusion describes TDH as an end-to-end framework combining feature learning, hash-code learning, triplet supervision, inter- and intra-modal losses, and graph regularization.Experiments on two popular datasets show that TDH outperforms several state-of-the-art approaches.
Loading 1904.02449v1…