PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Binary Reconstruction for Cross-modal Hashing

Xuelong Li, Di Hu, Feiping Nie

arXiv:1708.05127v2cs.CVcs.MM

TL;DR

Cross-modal hashing seeks efficient retrieval across modalities, but binary constraints make direct optimization difficult and prior methods often threshold real-valued representations afterward. The paper analyzes MRBM consistency and introduces DBRC, which uses Adaptive Tanh to learn binary codes directly through back-propagation and reconstruction. Experiments on three benchmark datasets report better compact codes for both image2text and text2image retrieval.

  • Problem

    Existing cross-modal hashing methods commonly threshold real-valued representations because the binary constraint makes the hashing objective difficult to optimize directly.

  • Method

    DBRC analyzes MRBM consistency and directly learns binary codes through a multimodal reconstruction network using the differentiable, learnable Adaptive Tanh activation.

  • Results

    DBRC generates better compact codes than prior methods in both image2text and text2image retrieval across three benchmark datasets.

  • Takeaways & Limitations

    Adaptive Tanh enables binary-code learning within the network while retaining back-propagation-based optimization.

Abstract

from arXiv · show

With the increasing demand of massive multimodal data storage and organization, cross-modal retrieval based on hashing technique has drawn much attention nowadays. It takes the binary codes of one modality as the query to retrieve the relevant hashing codes of another modality. However, the existing binary constraint makes it difficult to find the optimal cross-modal hashing function. Most approaches choose to relax the constraint and perform thresholding strategy on the real-value representation instead of directly solving the original objective. In this paper, we first provide a concrete analysis about the effectiveness of multimodal networks in preserving the inter- and intra-modal consistency. Based on the analysis, we provide a so-called Deep Binary Reconstruction (DBRC) network that can directly learn the binary hashing codes in an unsupervised fashion. The superiority comes from a proposed simple but efficient activation function, named as Adaptive Tanh (ATanh). The ATanh function can adaptively learn the binary codes and be trained via back-propagation. Extensive experiments on three benchmark datasets demonstrate that DBRC outperforms several state-of-the-art methods in both image2text and text2image retrieval task.

1 INTRODUCTION

Cross-modal hashing addresses efficient retrieval across modalities, but existing unsupervised methods typically learn real-valued shared representations and threshold them afterward. DBRC instead analyzes multimodal consistency and directly learns binary codes through an adaptive activation and reconstruction framework.

  • Cross-modal retrieval uses one modality as a query for relevant data in another, but traditional approaches become inefficient with massive multimodal collections.
  • Hashing reduces storage and retrieval costs by replacing high-dimensional real-valued data with short binary codes.
  • Existing unsupervised cross-modal hashing methods commonly generate shared real-valued codes and then binarize them through thresholding.
  • Deep multimodal methods can model nonlinear cross-modal relationships, but prior approaches often combine conventional networks with post hoc binarization rather than directly learning hashing codes.
  • DBRC analyzes MRBM consistency, directly learns binary codes with Adaptive Tanh, and reconstructs multimodal data in an unsupervised fashion.The hashing layer is trained through back-propagation within the deep network.

2 RELATED WORK

Related work groups unsupervised cross-modal hashing into linear and nonlinear approaches that project modalities into a shared space before thresholding. Deep multimodal networks address nonlinear relationships through modality-specific networks connected by a shared layer.

  • Existing unsupervised cross-modal hashing methods project different modalities into a common low-dimensional space and then threshold the projected real-valued vectors.
  • Linear modeling: Linear methods learn the common subspace through linear projections, which limits their effectiveness in modeling shared representations.
  • Nonlinear modeling: Nonlinear methods use modality-specific deep networks and a shared layer to learn joint multimodal representations.MDAE introduced this shared-layer structure for multimodal learning, and related frameworks were later used for retrieval and hashing.

3 MULTIMODAL MAXIMUM LIKELIHOOD LEARNING

MRBM models the joint distribution of two modalities and shared hidden units, with maximum likelihood learning decomposing into unimodal and cross-modal components. This decomposition supports preservation of both intra-modal and inter-modal consistency.

  • Cross-modal hashing must preserve both inter-modal and intra-modal correlations, motivating multimodal networks that fuse multiple modalities.
  • MRBM defines a joint distribution over modality x, modality y, and shared hidden units h.It is an energy-based network and a special case of a restricted Boltzmann machine.
  • The model marginalizes the shared hidden units h to obtain the joint likelihood P(x, y).
  • The MRBM joint likelihood is optimized with Contrastive Divergence or Persistent Contrastive Divergence to approximate its gradient.
  • Maximum likelihood learning of MRBM simultaneously models unimodal and cross-modal data distributions, preserving intra-modal and inter-modal consistency.

4 DEEP BINARY RECONSTRUCTION

DBRC directly learns cross-modal binary codes by combining adaptive tanh activation with multimodal reconstruction. ATanh approaches sign-like binary outputs while remaining differentiable, and the framework uses shared training codes with modality-specific testing models.

  • 4.1 Adaptive Tanh: ATanh replaces post-training thresholding with a learnable activation that can approach binary hashing values while remaining differentiable for back-propagation.A small α makes ATanh behave like tanh, whereas a sufficiently large α makes it approach sign.
  • 4.1 Adaptive Tanh: The ATanh regularization term controls α's magnitude, making the adaptive function more reliable for binary code learning.The paper notes that larger λ pushes hashing units closer to binary values early but can damage abstract multimodal representations and slow convergence.
  • 4.1 Adaptive Tanh: ATanh is element-wise, allowing different scaling parameters for different bits rather than enforcing one shared sign-like transformation.The paper gives 32 separate ATanh functions as an example for a 32-bit code.
  • 4.2 Optimization: Each ATanh parameter α_i is updated by back-propagation through the chain-rule derivative of the reconstruction objective.The update can use stochastic gradient descent with RMSprop; the paper initializes α_i = 1 and reports negligible ATanh computation time.
  • 4 DEEP BINARY RECONSTRUCTION: DBRC encodes each modality into low-dimensional representations, learns a joint representation through MRBM, and reconstructs the original data from shared binary codes.The multimodal deep reconstruction network is designed to directly generate cross-modal hashing codes.
  • 4 DEEP BINARY RECONSTRUCTION: Training uses both modalities to learn identical hashing codes, while testing uses different modality-specific models because retrieval queries provide only one modality.The paper separately describes hashing-code procedures for training and testing data.

5 EXPERIMENTS

Experiments evaluate DBRC on three benchmark datasets using image2text and text2image retrieval, varying code lengths and comparing hashing-ranking, hash-lookup, and activation variants. DBRC generally performs strongly, with direct binary learning addressing weaknesses of thresholded deep representations.

  • 5.1 Setup: Experiments evaluate image2text and text2image retrieval on Wiki, FLICKR-25K, and NUS-WIDE with varying code lengths.Hamming ranking and hash lookup are both used, with MAP for ranking and a Hamming ball of radius 2 for lookup.
  • 5.1 Setup: The evaluation compares DBRC with four linear and two nonlinear unsupervised cross-modal hashing baselines.The nonlinear baselines are Corr-Full-AE and DMHOR; the linear baselines are IMH, CVH, CMFH, and LSSH.
  • 5.3 Experimental Results: DBRC shows remarkable MAP performance on Wiki, while thresholded deep methods can underperform conventional linear modeling because of unbalanced hidden activations.DBRC directly learns binary codes from reconstruction rather than thresholding hidden units.
  • 5.3 Experimental Results: DBRC achieves the best Hamming-ranking performance on FLICKR-25K and the best MAP scores on NUS-WIDE for both image2text and text2image retrieval.On FLICKR-25K, DBRC remains substantially superior as code length increases, although CMFH has higher precision; DBRC offers a better precision-recall balance.
  • 5.3 Experimental Results: Increasing λ pushes hashing-layer units closer to binary values early in training, which can damage cross-modal abstract representations and slow convergence.The experiments use λ = 0.001 across datasets.
  • 5.3 Experimental Results: Learnable ATanh variants outperform the fixed-sequence variant because they adapt the binarization function to the projected subspace and individual bits.The comparison is reported in Table 2 for Wiki Hamming-ranking MAP across code lengths.
  • 5.3 Experimental Results: The hash-lookup evaluation reports Precision and F-measure across Wiki, FLICKR-25K, and NUS-WIDE for varying code lengths.The paper uses hash lookup to focus on top retrieved items within a specific Hamming ball.

6 CONCLUSION

The paper proposes directly learning cross-modal hashing codes by reconstructing original data from a shared binary representation. DBRC uses ATanh for adaptive, back-propagation-trainable binarization and achieves better compact-code retrieval results across both retrieval directions on three benchmark datasets.

  • 6 CONCLUSION: DBRC directly learns cross-modal hashing codes by reconstructing original data from an embedded shared binary representation.This differs from methods that first learn real-valued representations and then threshold them.
  • 6 CONCLUSION: ATanh adaptively learns binary codes within the network and can be trained via back-propagation.The activation function is presented as the source of DBRC’s ability to optimize binary codes directly.
  • 6 CONCLUSION: Experiments on three benchmark datasets show better compact codes for both image2text and text2image retrieval.The conclusion summarizes the reported advantage over previous cross-modal hashing approaches.
Loading 1708.05127v2…