PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on Cross-modal Retrieval

Kaiye Wang, Qiyue Yin, Wei Wang, Shu Wu, Liang Wang

arXiv:1607.06215v1cs.MMcs.CLcs.IR

TL;DR

Rapidly growing multimodal data creates a need to retrieve relevant content across heterogeneous modalities while measuring their semantic similarity. The paper surveys and taxonomizes representative cross-modal retrieval methods, datasets, and evaluations, distinguishing real-valued and binary representations. Its comparisons reveal characteristics of different methods, while the discussion identifies open problems and future directions.

  • Problem

    Cross-modal retrieval must measure content similarity between heterogeneous modalities so one data type can retrieve relevant data of another type.

  • Method

    The paper surveys representative methods, classifies them into real-valued and binary representation learning, introduces multimodal datasets, evaluates algorithms, and discusses future trends.

  • Results

    Comparisons on commonly used datasets reveal characteristics of different cross-modal retrieval methods.

  • Takeaways & Limitations

    The survey is intended to support understanding of cross-modal retrieval algorithms, practical applications, and future research.

  • Takeaways & Limitations

    Existing multimodal datasets have limitations: Wiki is small, NUS-WIDE uses simple user tags, and Flickr30K lacks category information.

Abstract

from arXiv · show

In recent years, cross-modal retrieval has drawn much attention due to the rapid growth of multimodal data. It takes one type of data as the query to retrieve relevant data of another type. For example, a user can use a text to retrieve relevant pictures or videos. Since the query and its retrieved results can be of different modalities, how to measure the content similarity between different modalities of data remains a challenge. Various methods have been proposed to deal with such a problem. In this paper, we first review a number of representative methods for cross-modal retrieval and classify them into two main groups: 1) real-valued representation learning, and 2) binary representation learning. Real-valued representation learning methods aim to learn real-valued common representations for different modalities of data. To speed up the cross-modal retrieval, a number of binary representation learning methods are proposed to map different modalities of data into a common Hamming space. Then, we introduce several multimodal datasets in the community, and show the experimental results on two commonly used multimodal datasets. The comparison reveals the characteristic of different kinds of cross-modal retrieval methods, which is expected to benefit both practical applications and future research. Finally, we discuss open problems and future research directions.

1 INTRODUCTION

Cross-modal retrieval addresses the need to search across heterogeneous media when traditional methods mainly support same-modality retrieval. This survey reviews recent methods, evaluates representative algorithms, and identifies challenges and future directions.

  • Traditional retrieval techniques mainly search within one media type, creating a need for models that support multimodal similarity search.
  • Cross-modal retrieval lets users submit one modality, such as a photo, to retrieve relevant content in another modality, such as text.
  • The central challenge is measuring content similarity across modalities, known as the heterogeneity gap, while improving retrieval accuracy and scalability.
  • The paper surveys recent progress and provides a taxonomy explaining differences among cross-modal retrieval approaches.
  • It evaluates representative algorithms on commonly used datasets and reports findings intended to support understanding, applications, and future research.
  • The survey also summarizes challenges and opportunities and identifies open directions for future work.

2 OVERVIEW

Cross-modal retrieval methods are organized by their learned representation and the information used during learning. Binary methods accelerate search in a common Hamming space, but encoding can slightly reduce accuracy.

  • The general retrieval framework extracts modality-specific features, models cross-modal correlations, learns common representations, and retrieves across modalities.
  • Approaches are divided into real-valued representation learning and binary representation learning, also called cross-modal hashing.
  • Binary methods transform modalities into a common Hamming space, enabling fast cross-modal similarity search.
  • Binary encoding generally decreases retrieval accuracy slightly because information is lost during conversion to codes.
  • Methods are further grouped as unsupervised, pairwise based, rank based, or supervised according to the information used for learning.
  • Unsupervised methods use co-occurrence, pairwise methods use similar or dissimilar pairs, rank based methods use rank lists, and supervised methods use labels.
  • Supervised methods seek more discriminative representations by separating different-class samples and bringing same-class samples together, but labels can be expensive to obtain.

3 REAL-VALUED REPRESENTATION LEARN-

Real-valued representation learning seeks a shared real-valued space where data from different modalities can be directly compared. Methods are categorized by the information used to learn that common representation.

  • Real-valued methods learn a common representation space in which relevant data from different modalities are close and cross-modal similarity can be directly measured.

3.1 Unsupervised methods

Unsupervised methods learn common representations across modalities primarily from co-occurrence information, without explicit similarity labels. They include subspace learning, topic models, and deep learning approaches.

  • Overview: Unsupervised methods use co-occurrence information to learn common representations across modalities.Modalities co-existing in a multimodal document are treated as semantically related.
  • Subspace learning: Subspace learning methods map different modalities into a shared latent space where cross-modal similarity can be measured.They enforce pairwise closeness between corresponding multimodal data.
  • Subspace learning: CCA learns directions for two modalities along which their data are maximally correlated.It has been applied to cross-media, cross-lingual, and vision retrieval problems.
  • Subspace learning: Two-stage crossmodal retrieval first uses CCA to learn a common subspace, then learns a semantic space for measuring multimodal similarity.The first stage maximizes correlation between the modalities.
  • Topic models: Topic models represent cross-modal correlations with shared or related latent topics, including Corr-LDA, Tr-mm LDA, and MDRF.Corr-LDA uses shared topics, while Tr-mm LDA uses separate topic sets linked by regression; MDRF models document-level similarities.
  • Deep learning methods: Deep methods learn nonlinear or jointly optimized representations for multimodal data, including DCCA, end-to-end DCCA, correspondence autoencoders, and video-text embeddings.End-to-end DCCA propagates gradients through two network branches, while Corr-AE jointly minimizes modality-specific representation errors and cross-modal correlation errors.

3.2 Pairwise based methods

Pairwise-based methods learn cross-modal metric spaces from similar or dissimilar pairs. Their approaches range from projection-based shallow models to graph-regularized and latent-feature methods.

  • Overview: Pairwise-based methods use similar or dissimilar pairs to learn meaningful distances between modalities.They are treated as heterogeneous metric learning because the learned metric spans different feature spaces.
  • Shallow methods: Multi-NPP projects different modalities into a shared feature space where Euclidean distance measures intra- and inter-modality similarity.The method learns modality-specific projection functions.
  • Shallow methods: The Multi-NPP loss pulls similar objects together and pushes dissimilar objects apart in the latent space.Its objective combines a similarity term with a dissimilarity term.
  • Shallow methods: MVML-GL first seeks a globally consistent shared latent space, then learns explicit mappings from input spaces using regularized local linear regression.JGRHML additionally learns a high-level semantic metric through label propagation.
  • Deep methods: MSDS uses separate convolutional networks for images and texts to map raw data into latent representations that form related multimodal features.The text network includes word-embedding learning.

3.3 Rank based methods

Rank-based methods formulate cross-modal retrieval as learning to rank, using ranking preferences or objectives to organize retrieved items. Deep variants learn common spaces at global or fine-grained levels.

  • Overview: Rank-based methods use rank lists to learn common representations for cross-modal retrieval.They treat retrieval as a learning-to-rank problem.
  • Shallow methods: PAMIR adapts the Passive-Aggressive algorithm to cross-modal ranking, while SSI addresses cross-lingual retrieval.These methods represent shallow ranking-based approaches.
  • Shallow methods: Joint image-annotation models learn a low-dimensional embedding while optimizing precision at the top of ranked annotation lists.The embedding contains both images and annotations.
  • Shallow methods: LSCMR uses a structural SVM to learn a metric whose query-induced rankings can be optimized against ranking measures.The method does not fully use bidirectional ranking examples.
  • Shallow methods: RCCA adjusts a CCA-learned subspace to preserve preference relations in click data using a query-image similarity function and margin ranking loss.The margin loss contrasts positive and negative images for a query.
  • Deep learning methods: Deep ranking models embed images and sentences into shared spaces, with some aligning object and sentence fragments or jointly modeling local and global alignment.DT-RNNs compose sentence vectors, while C2MLR optimizes pairwise ranking with local and global alignment.
  • Deep learning methods: CMCDCA combines visual feature representation learning with cross-modal correlation learning under a large-margin principle.

3.4 Supervised methods

Supervised cross-modal retrieval methods use labels or semantic information to learn more discriminative shared representations, spanning subspace, feature, topic, and deep neural approaches.

  • Label information separates classes more effectively in the common representation space.
  • Supervised subspace methods push different-class samples apart while keeping same-class samples close.Several methods extend CCA with supervised objectives, including GMA, GMLDA, and GMMFA.
  • Cluster-CCA learns low-dimensional representations that maximize cross-modal correlation while segregating different classes.It is formulated for joint dimensionality reduction of two modalities.
  • CCA-3V incorporates visual, textual, and semantic-class views to model their dependence in a shared representation.The learned projections correspond to visual, text, and semantic class views.
  • Other supervised approaches include common discriminant features, manifold alignment, coupled feature spaces, sparse coding, topic models, and multimodal neural networks.These methods respectively exploit scatter structure, vector fields, coupled regressions, dictionaries, latent topics, or modality-specific mappings.
  • Deep methods map visual and textual inputs into common semantic spaces using multilayer networks, label supervision, aligned intermediate representations, or shared probability vectors.Examples include RE-DNN, multilabel deep learning, cross-modal convolutional networks, and MDNN.

4 BINARY REPRESENTATION LEARNING

Binary representation learning, or cross-modal hashing, addresses the cost of linear search by mapping multiple modalities into a shared Hamming space for fast retrieval.

  • Most real-valued methods rely on time-consuming brute-force linear search, motivating binary representation learning for large-scale data.
  • Cross-modal hashing discovers inter-modal correlations and projects different modalities into a common Hamming space for fast retrieval.Methods are categorized as unsupervised, pairwise-based, or supervised.
  • Early linear methods preserve cross-view similarity through spectral extensions, latent factors, projections, sparse coding, or inter- and intra-modal consistency.Representative methods include CVH, IMH, LCMH, CMFH, and LSSH.
  • Hashing objectives may jointly model intra-modal consistency, inter-modal consistency, and hash-function learning for new data.
  • Nonlinear hashing methods capture more complex multimodal structure using stacked autoencoders, orthogonal regularization, or multilayer mappings.These designs target accurate, compact codes while modeling intra- and inter-modality correlations.
  • SePH converts semantic affinities into a probability distribution, approximates it with Hamming codes using KL-divergence, and learns nonlinear view-specific projections.It also combines predicted codes and output probabilities for unseen instances.

5 MULTIMODAL DATASETS

The survey reviews five commonly used multimodal datasets, mostly pairing images with text, while noting limitations in scale, categories, text richness, and supervision.

  • The survey introduces Wikipedia, INRIA-Websearch, Flickr30K, Pascal VOC, and NUS-WIDE as commonly used multimodal datasets.
  • The Wiki dataset contains 2866 image-text pairs across 10 semantic classes, with articles describing people, places, or events.
  • INRIA-Websearch contains 71,478 image-text pairs spanning 353 concepts, making it challenging because of its large number of classes.
  • Flickr30K contains 31,783 images, each paired with five independently written sentences, and focuses on events involving people and animals.
  • NUS-WIDE contains 186,577 labeled images paired with user tags, commonly using the largest 10 or 21 classes for experiments.Its text modality is user tags, which provide relatively simple and limited descriptions.
  • Pascal VOC provides 5011 training and 4952 testing image-tag pairs across 20 classes, often reduced to single-object images for evaluation.The reduced setup contains 2808 training and 2841 testing examples.

6 EXPERIMENTS

The experiments compare representative real-valued and binary cross-modal retrieval methods across image-text and image-sentence tasks. Results generally favor supervised, discriminative, semantically informed, nonlinear, and modality-appropriate representations.

  • Image-text retrieval: Supervised methods generally outperform unsupervised methods because class information improves separation in the common representation space.The comparison includes CDFE, GMMFA, GMLDA, CCA-3V, SliM2, M3R, LCFS, and JFSSL versus PLS, BLM, and CCA.
  • Image-text retrieval: JFSSL outperforms other methods in all three-modality Wiki retrieval tasks by modeling correlations across N modalities.Two-modality methods are not suitable for the simulated three-modality setting.
  • Image-sentence retrieval: End-to-end DCCA and Deep Fragment perform better than DeViSE and SDT-RNN on Flickr30K image-sentence retrieval.Deep Fragment explicitly aligns image objects with sentence dependency relations, while End-to-end DCCA models global visual and textual properties with implicit alignment.
  • Binary representation learning: IMH outperforms CVH because it uses both inter-modality and intra-modality similarity.The result supports modeling within-modality similarity in cross-modal hashing.
  • Binary representation learning: CMSSH outperforms CVH because it uses similar and dissimilar pairs rather than pairwise information alone.The experiments associate using more information with improved performance.
  • Binary representation learning: Supervised hashing methods improve as hash-code length increases, whereas some baselines decline, indicating that longer binary codes are harder to learn without supervision.SCM-Seq and SePH improve with longer codes, while CVH, IMH, and CMSSH show the contrasting pattern reported in the experiments.

7 DISCUSSION AND FUTURE TRENDS

The discussion identifies dataset quality, limited and noisy annotations, scalability, deep multimodal learning, and finer-grained correspondence as priorities for future cross-modal retrieval research. Existing benchmarks remain limited in size, modality coverage, or textual richness.

  • Open challenges: The paper notes a continuing gap between state-of-the-art performance and user expectations, leaving substantial room for further research.This motivates the future directions discussed in the section.
  • Datasets: Existing datasets are often too small or too specific, motivating large-scale multimodal benchmarks with more than two modalities and ground truth.The Wiki dataset contains 2866 documents, while NUS-WIDE primarily provides user tags.
  • Annotations: Web multimodal data contain limited and noisy annotations, making large-scale semantic supervision difficult to obtain.The discussion highlights social-network data as a major source of loosely organized multimodal content.
  • Scalability: Future methods must scale to massive multimodal collections and distributed platforms while organizing related modalities efficiently.The need follows the rapid growth and propagation of multimedia resources on the web.
  • Deep learning: Deep learning is identified as a future direction because powerful representations can reduce heterogeneity and semantic gaps between modalities.The paper specifically points to CNNs for images and RNNs for text.

5. Finer-level cross-modal semantic correlation modeling

Most methods embed modalities into a shared space, but image-text retrieval also requires modeling correspondences between finer-grained fragments. The paper calls for models that capture these complex relations more accurately.

  • Finer-level correspondence: Shared embedding spaces can be too coarse because different image fragments may correspond to different text fragments.Modeling fragment-level correspondence is presented as a way to capture image-text semantic relations more accurately.
  • Finer-level correspondence: Future models should obtain modality-specific fragments and learn their correspondences to represent complex cross-modal relations.The paper identifies fragment extraction and correspondence modeling as important design problems.

8 CONCLUSIONS

The paper surveys cross-modal retrieval methods, organizes them into real-valued and binary representation learning, evaluates representative algorithms and datasets, and discusses future trends. It concludes that the field remains insufficiently addressed and needs further work.

  • Contributions: The survey classifies representative methods into real-valued representation learning and binary representation learning.It also introduces datasets and empirically evaluates representative methods.
  • Conclusion: The paper aims to help readers understand the state of the art and motivate further research in cross-modal retrieval.Its scope includes methods, datasets, experiments, and future trends.
Loading 1607.06215v1…