PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions

Tianshi Wang, Fengling Li, Lei Zhu, Jingjing Li, Zheng Zhang, Heng Tao Shen

arXiv:2308.14263v3cs.IRcs.MM

TL;DR

Cross-modal retrieval serves users seeking information across modalities, while earlier reviews have been limited in timeliness, taxonomy, and coverage. This paper synthesizes methods through a detailed taxonomy, architectural review, evaluation resources, and a toolbox. Across reviewed evaluations, supervised methods generally achieve higher accuracy, deep methods generally outperform shallow methods, and Transformer and VLP methods lead real-valued retrieval results.

  • Problem

    Earlier cross-modal retrieval reviews have limitations in timeliness, taxonomy, and comprehensiveness, despite growing demand for access to data across modalities.

    Source: §I · pp. 1–2

  • Method

    The paper systematically reviews more than three hundred articles, develops a nuanced taxonomy, explains methods and architectures, and compiles evaluation resources and a user-friendly toolbox.

    Source: §I · p. 2

  • Results

    Supervised methods generally achieve higher retrieval accuracy than unsupervised methods, deep models generally outperform shallow methods, and Transformer and VLP methods lead real-valued retrieval accuracy.

    Source: C. Evaluation Result · p. 24

  • Takeaways & Limitations

    Researchers can use the compiled datasets, metrics, and toolbox to choose experimental paradigms and implement cross-modal retrieval.

    Source: §I · p. 2

Abstract

from arXiv · show

With the exponential surge in diverse multi-modal data, traditional uni-modal retrieval methods struggle to meet the needs of users seeking access to data across various modalities. To address this, cross-modal retrieval has emerged, enabling interaction across modalities, facilitating semantic matching, and leveraging complementarity and consistency between heterogeneous data. Although prior literature has reviewed the field of cross-modal retrieval, it suffers from numerous deficiencies in terms of timeliness, taxonomy, and comprehensiveness. This paper conducts a comprehensive review of cross-modal retrieval's evolution, spanning from shallow statistical analysis techniques to vision-language pre-training models. Commencing with a comprehensive taxonomy grounded in machine learning paradigms, mechanisms, and models, the paper delves deeply into the principles and architectures underpinning existing cross-modal retrieval methods. Furthermore, it offers an overview of widely-used benchmarks, metrics, and performances. Lastly, the paper probes the prospects and challenges that confront contemporary cross-modal retrieval, while engaging in a discourse on potential directions for further progress in the field. To facilitate the ongoing research on cross-modal retrieval, we develop a user-friendly toolbox and an open-source repository at https://cross-modal-retrieval.github.io.

I. INTRODUCTION

Cross-modal retrieval addresses the need to search across heterogeneous data, while this review responds to gaps in earlier reviews with a broad taxonomy and synthesis of methods, evaluation resources, and future directions. Its overview organizes methods by representation, supervision, and scenario, while noting trade-offs between retrieval approaches.

I. INTRODUCTION

  • Earlier reviews were limited in timeliness, taxonomy, and coverage, particularly for Transformer and vision-language pre-training methods, broader modalities, and practical scenarios.The paper reviews more than three hundred articles to provide a broad account of the field.Source: §I · p. 2
  • The review offers detailed accounts of cross-modal retrieval principles and architectures, spanning fundamental concepts and newer developments.It also examines opportunities, challenges, and potential research directions.Source: §I · p. 2
  • The paper introduces a user-friendly toolbox and compiles datasets, metrics, and benchmarks to support implementation and experimental selection.These resources are intended to help researchers choose paradigms for validation and evaluation.Source: §I · p. 2

II. OVERVIEW

  • Real-value retrieval retains richer semantic information but costs more storage and computation, whereas hashing compresses data for more efficient retrieval at some semantic-information cost.The choice depends on application requirements and constraints.Source: §II · p. 3
  • The overview classifies methods into five categories: supervised or unsupervised real-value retrieval, supervised or unsupervised hashing retrieval, and retrieval under special scenarios.The taxonomy is further refined by technical architectures or scenarios, with forty-four subcategories stated in the introduction.Source: §I · p. 2, §II · p. 3
  • The taxonomy refines the first four categories by statistical methods and network architectures, while special-scenario retrieval includes constrained settings such as noisy, incomplete, incremental, and cross-domain data.The framework is presented as applicable to combinations including text-image, text-video, text-audio, and image-audio retrieval.Source: §II · pp. 3–4

III. TEXT-IMAGE CROSS-MODAL RETRIEVAL

The section organizes text-image cross-modal retrieval methods by retrieval objective, learning approach, and architecture. It formalizes image, text, label, and similarity notation before surveying unsupervised real-value methods, including CCA and topic learning.

III. TEXT-IMAGE CROSS-MODAL RETRIEVAL

  • Text-image retrieval methods are catalogued across five categories, with shared notation for image, text, labels, and pairwise category similarity.The review also distinguishes symbols unavailable to unsupervised retrieval: labels and the similarity matrix.Source: §III · pp. 4–5

A. Unsupervised Real-value Retrieval

  • Unsupervised real-value retrieval separates category-semantic retrieval from direct image-text matching, which use different objectives and evaluation metrics.Category-semantic methods retrieve items sharing query categories and use measures such as mAP; image-text matching targets corresponding samples and uses measures such as Recall.Source: A. Unsupervised Real-value Retrieval · p. 5

1) Canonical correlation analysis methods:

  • CCA maps paired image and text features into a common low-dimensional space by learning linear transformations that maximize correlation.Later KCCA and DCCA methods address CCA’s limits in modeling nonlinear relationships; MCCA supports multiple views.Source: 1) Canonical correlation analysis… · p. 6

2) Topic learning methods:

  • Topic-learning methods represent multimodal data with topic and modality-specific distributions, then model cross-modal relations through shared latent topics or other linking mechanisms.Corr-LDA uses conditional generation; tr-mmLDA adds regression, while MDRF uses a Markov random field to relate topics across modalities.Source: 2) Topic learning methods · p. 6

3) Self-reconstructing methods:

The reviewed methods develop cross-modal image-text representations through reconstruction, neural feature matching, graph reasoning, Transformers, and vision-language pretraining. Their mechanisms include preserving cross-modal distances, aligning fine-grained content, and using pretrained representations.

3) Self-reconstructing methods:

  • Self-reconstructing methods learn similar multimodal representations with autoencoders, coordinating within-modality reconstruction and cross-modal relationships.The surveyed approaches include stacked autoencoders and denoising with a distance-preserving common space and joint loss.Source: 3) Self-reconstructing methods · p. 6

4) CNN-RNN methods:

  • CNN-RNN methods extract modality-specific features and use paired positive and negative samples to constrain cross-modal matching.Approaches address hard-sample ranking, rare-content representation, one-to-many correspondence, and fine-grained spatial attention.Source: 4) CNN-RNN methods · pp. 6–7

5) GNN methods:

  • Image-text GNN methods construct object graphs within each multimodal sample, then use graph networks to align the modalities.This differs from other cross-modal retrieval tasks, which construct neighbor graphs between samples.Source: 5) GNN methods · p. 7, Figure 5 · p. 7

6) Transformer methods:

  • Transformer methods improve retrieval through residual learning, probabilistic embeddings, multi-level alignment, and multi-granularity matching.The surveyed approaches also model intra- and inter-modal relations and use contrastive losses, including with weakly positive pairs.Source: 6) Transformer methods · p. 7

7) VLP model methods:

  • VLP retrieval methods use single- or dual-stream pretrained architectures and refine alignment through training, fine-tuning, re-ranking, and token-level interaction.Examples include aligning visual patches with words and combining contrastive learning with keyword enhancement for E-commerce retrieval.Source: 7) VLP model methods · pp. 7–8

8) CMG model methods:

The review contrasts cross-modal generation methods with supervised real-value retrieval, spanning shallow statistical approaches and deep networks. It outlines how supervised methods use labels to model cross-modal associations, including through correlation and dictionary learning.

8) CMG model methods:

  • Cross-modal generation adds generation tasks to retrieval training, aligning generated samples with retrieval results while building on existing image-text matching methods.The review describes this as multi-task learning that indirectly optimizes retrieval models.Source: 8) CMG model methods · p. 8

B. Supervised Real-value Retrieval

  • Supervised real-value retrieval uses labels to model semantic associations; shallow methods include five statistical-learning families, while deep methods include CNN-RNN, GAN, GNN, and Transformer approaches.The shallow families are CCA, dictionary, projection, topic, and metric learning.Source: B. Supervised Real-value Retrieval · pp. 8–9

1) Canonical correlation analysis methods:

  • Supervised CCA learns modality transformations that maximize correlation between projected features, with label information used to establish cross-modal correspondence.GMA seeks a nonlinear subspace, while ml-CCA uses multi-label annotations without requiring explicit paired modalities.Source: 1) Canonical correlation analysis… · p. 9

2) Dictionary learning methods:

  • Discriminative Dictionary Learning learns modality-specific representations and maps sparse codes into a unified label space.The methods aim to capture shared-modality structure and inter-modal relationships.Source: 2) Dictionary learning methods · pp. 9–10

3) Projection learning methods:

The review surveys supervised real-value approaches and begins its account of unsupervised hashing, covering projection, topic, metric, and neural methods alongside matrix factorization. The methods use distinct mechanisms to represent cross-modal relationships or produce hash codes.

3) Projection learning methods:

  • Projection learning maps modalities into a shared low-dimensional space; JFSSL uses label information, feature selection, and graph regularization, while MDCR learns distinct projection spaces for different retrieval tasks.JFSSL’s feature selection uses an l2,1-norm penalty.Source: 3) Projection learning methods · p. 10

4) Topic learning methods:

  • Topic-learning methods represent modalities in a unified semantic space, using joint probabilistic modeling or supervised topic models to capture cross-modal and within-modal structure.M3R models mutually consistent topics; NPBUS incorporates shared supervised responses into a hierarchical Dirichlet process.Source: 4) Topic learning methods · p. 10

5) Metric learning methods:

  • Metric learning creates a discriminative shared space by bringing semantically similar cross-modal features closer and separating dissimilar ones through pairwise or triplet losses.PFAR aligns parallel fields, while PL-ranking uses pairwise ranking losses and class information.Source: 5) Metric learning methods · p. 10

6) CNN-RNN methods:

  • CNN-RNN methods learn shared representations with neural networks; DSCMR uses label constraints and pairwise loss, while SDCMR uses dual adversarial mechanisms to isolate shared semantics.Source: 6) CNN-RNN methods · p. 10

7) GAN methods:

  • GAN methods use adversarial training to encourage more discriminative real-value features, with AAL adaptively balancing loss contributions based on data dispersion.ACMR uses a feature projector and modality classifier in a minimax game with triplet constraints.Source: 7) GAN methods · p. 10

8) GNN methods:

  • GNN methods exploit graph topology to capture neighboring semantic information; supervised approaches incorporate labels, and WGSCN models modality-specific and shared correlations.SSPE preserves local graph structure and reconstructs labels for global semantic retention.Source: 8) GNN methods · pp. 10–11

9) Transformer methods:

  • Transformer methods use multi-head attention to encode multimodal relationships; RLCMR combines modality tokens in one encoder, while IEFT models text-image pairs as unified entities.Source: 9) Transformer methods · p. 11

C. Unsupervised Hashing Retrieval

2) Spectral graph learning methods:

The review describes unsupervised hashing methods that learn binary codes from multimodal data through graph, metric, quantization, neural, adversarial, and graph-network approaches. Their mechanisms range from preserving neighbor relationships to reconstructing data or transferring semantic structure.

2) Spectral graph learning methods:

  • Spectral graph methods represent multimodal samples as graph vertices and relationships as edges, using modality-specific neighbor graphs and Laplacian operators to support hash learning.LCMH uses anchor graphs to preserve within-modality similarity; SM2H uses hyper-graphs to capture within- and cross-modality similarity.Source: 2) Spectral graph learning methods · p. 12

3) Metric learning methods:

  • Metric-learning methods preserve multimodal relationships in hash codes; PDH gives similar samples identical codes, while IMVH jointly optimizes within-view similarity and cross-view correlation.UDC uses tail codes for content similarity and head codes for semantic guidance.Source: 3) Metric learning methods · p. 12

4) Quantization learning methods:

  • Quantization methods directly learn binary hash codes, minimizing distances between cross-modal pairs, their codes, and binary-hypercube vertices.CCQ also maps modalities into a correlated, isomorphic potential space before producing compact binary codes.Source: 4) Quantization learning methods · p. 12

5) CNN-RNN methods:

  • Deep unsupervised hashing uses neural networks and regularization to generate codes; DBRC adaptively binarizes joint representations, while UCHM builds codes through modality interaction.UDCMH preserves neighbor structure with Laplacian constraints; UCHSTM mines text correlations to guide network training.Source: 5) CNN-RNN methods · p. 12

6) GAN methods:

  • GAN methods reconstruct one modality using information from another to strengthen semantic association and generate robust hash codes.UGACH challenges a discriminator with informative generated samples; UCH uses two-cycle GANs for common representations and hash codes.Source: 6) GAN methods · p. 12

7) GNN methods:

  • GNN methods use modality-specific graphs and graph networks to align multimodal data; DJSRH reconstructs a joint-semantic affinity graph, while CMGCH uses cross-modal contrastive attention.CMGCH also learns multi-granularity embeddings to improve representation discriminativeness and consistency.Source: 7) GNN methods · pp. 12–13

8) Transformer methods:

The review moves from emerging Transformer and distillation approaches to supervised hashing, which uses labels to learn compact codes for efficient search. It organizes supervised methods by architecture and by the constraints used to map data into Hamming spaces.

8) Transformer methods:

  • Unsupervised hashing Transformers use self-attention to model cross-modal dependencies, with HuggingHash aligning global hash codes and fine-grained content tokens.UCMFH combines multimodal fusion with contrastive learning to produce informative, compact hash codes.Source: 8) Transformer methods · p. 13

9) Distillation network methods:

  • Distillation methods train an unsupervised teacher to form a similarity matrix that guides a student model’s hashing; CKDH transfers CLIP similarity measurements and semantic relevance to a lightweight student.The review reports that CKDH improves retrieval accuracy and scalability.Source: 9) Distillation network methods · pp. 13–14

D. Supervised Hashing Retrieval

  • Supervised hashing uses manual annotations to map multimodal data into low-dimensional Hamming space; shallow methods use matrix factorization or projection, while deep methods learn codes end-to-end with neural networks.Its deep-method taxonomy includes CNN-RNN, GAN, GNN, Transformer, memory-network, and quantization-network approaches.Source: Figure 9 · p. 13, D. Supervised Hashing Retrieval · p. 14

1) Matrix factorization methods:

  • Supervised matrix factorization incorporates labels into a Laplacian constraint on shared hash codes, while dictionary methods use category side information and can constrain code similarity directly.SMFH extends unsupervised CMFH with label information; DLCMH uses a label-similarity matrix to improve code discriminability.Source: 1) Matrix factorization methods · p. 14

2) projection learning methods:

  • Supervised projection methods learn mappings into lower-dimensional Hamming spaces under label, separate-Hamming, common-Hamming, or relation constraints.Relation constraints preserve neighbor or ranking relationships; label and common-space approaches use labels to improve hash-code discriminability.Source: 2) projection learning methods · pp. 14–15

3) CNN-RNN methods:

CNN-RNN supervised hashing methods use neural networks to learn cross-modal representations under different training constraints. These constraints shape how methods preserve similarity, semantic relationships, and compact codes.

3) CNN-RNN methods:

  • The comparison organizes CNN-RNN methods by constraint type, including distance, similarity, negative log-likelihood, and triplet constraints.The displayed core losses omit some presentation details; modality-specific networks, reconstructed matrices, and triplet-loss margins are described in the accompanying notes.Source: Table III · p. 15, 3) CNN-RNN methods · p. 15
  • Distance-constrained methods learn cross-modal approximations or reconstruct features; MM-NN uses a coupled Siamese network, while CAH separates categories and reduces within-category variance.MM-NN manages intra- and inter-modal similarity measures.Source: 3) CNN-RNN methods · p. 15
  • Similarity-constrained methods align hash-code similarity with category-label similarity; DVSH aligns cross-modal relationships, while MRDH also separates dissimilar codes.MRDH uses pair-wise similarity consistency and a global similarity metric.Source: 3) CNN-RNN methods · p. 15
  • NLL-based methods use similarity matrices to supervise hashing; DCMH jointly extracts features and learns codes, while DCH-SCR supervises feature- and label-level rankings.DCH-SCR is described as capturing multi-level semantics and narrowing the modality gap.Source: 3) CNN-RNN methods · p. 15
  • Triplet-based methods select positive and negative samples from semantic labels and optimize ranking loss; TDH transfers similarity to Hamming space using graph regularization.SCH classifies sample pairs into semantic groups and applies tailored constraints in Hamming space.Source: 3) CNN-RNN methods · pp. 15–16

4) GAN methods:

The review groups methods by architecture and then turns to special retrieval settings, where incomplete data, noisy annotations, and task-specific needs motivate tailored approaches.

4) GAN methods:

  • GAN-based hashing uses adversarial and discriminative constraints; SDAH separates modality-shared from modality-private features and combines information bottleneck and structural similarity constraints.SSAH uses self-supervised semantic learning, while CPAH combines information decoupling with multi-task adversarial learning.Source: 4) GAN methods · p. 16

5) GNN methods:

  • Graph methods build modality-specific graphs to capture semantic relationships; PGCH uses proxy hash codes and graph convolution to reduce modality differences and improve retrieval.GCH uses a GCN to enhance correlation information; LGCNH reconstructs local graphs in modality features.Source: 5) GNN methods · p. 16

6) Transformer methods:

  • Transformer hashing captures fine-grained semantics and complex cross-modal relationships; DHaPH models semantic hierarchies with shared proxies and focuses dynamically on hard pairs.DCHMT adds a micro-hashing module, and MITH hierarchically models cross-modal semantic similarities.Source: 6) Transformer methods · p. 16

7) Memory network methods:

  • Memory networks use shared or independent modality memory banks; CMPD shares features between image and text generators and matches joint distributions.Memory networks are also used in few-shot and incremental retrieval settings.Source: 7) Memory network methods · p. 16

8) Quantization network methods:

  • Quantization networks convert real-valued features into compact binary codes; CDQ jointly learns deep features and quantizers, while ACQH uses compositional quantization.ACQH learns low-dimensional embeddings with projection matrices.Source: 8) Quantization network methods · p. 16

E. Cross-modal Retrieval under Special Scenarios

1) Noise-robust cross-modal retrieval:

  • Noise-robust methods address noisy labels and mismatched image-text pairs; DECL estimates pair uncertainty, while RDE identifies clean training data and uses triplet alignment.WSJE prioritizes clean data with curriculum learning; MRL combines clean-sample clustering and multi-modal contrastive loss; LCNME corrects labels using network predictions.Source: 1) Noise-robust cross-modal retrieval · p. 17

2) Incomplete cross-modal retrieval:

  • Incomplete-data methods learn from partial-missing or unpaired samples; RUCMH reconstructs objects in a common space without paired data, while CICH aligns contextual correspondences contrastively.TFNH uses zero-padding and data classifiers; PAN learns unified category prototypes and fuses them with nearest neighbors.Source: 2) Incomplete cross-modal retrieval · p. 17

3) Zero/few-shot cross-modal retrieval:

The review surveys retrieval methods for unseen classes, changing data, decentralized settings, composed queries, fine-grained semantics, and adversarial threats.

3) Zero/few-shot cross-modal retrieval:

  • Zero-shot methods transfer knowledge to unseen classes: TANSS uses word vectors for seen and unseen labels, while AG-MIH learns instance-level image hash codes.AG-MIH uses a category-attribute relation layer and multi-stream instance hashing refinement; the passage also reports noisy-label handling.Source: 3) Zero/few-shot cross-modal retrieval · p. 17
  • Few-shot methods use new-class semantics and cross-modal memory: Self-Others Net explores relationships among samples, while ACMM updates cross-modal prototypes.ACMM weakly supervises alignment between few-shot regions and words.Source: 3) Zero/few-shot cross-modal retrieval · p. 17

4) Incremental cross-modal retrieval:

  • Incremental retrieval integrates new data while preserving existing knowledge; CCMR avoids database re-indexing with updated networks, and CTP transfers embedding knowledge across tasks.The section identifies distribution shift and modality expansion as incremental-retrieval challenges.Source: 4) Incremental cross-modal retrieval · p. 17

5) Online cross-modal retrieval:

  • Online methods handle streaming data: ROH reduces pair-wise similarity factorization to linear optimization, while POLISH represents novel-category labels as new classes arrive.OCMFH dynamically updates existing hash codes as streaming data evolves.Source: 5) Online cross-modal retrieval · p. 17

6) Cross-domain cross-modal retrieval:

  • Cross-domain methods use labeled source-domain data to improve target-domain features; ACP aligns domain distributions and semantic commonalities, while CDTH transfers semantics from an auxiliary domain.DASG learns source-domain cross-modal correlations and aligns modalities and distributions.Source: 6) Cross-domain cross-modal retrieval · pp. 17–18

7) Federated cross-modal retrieval:

  • Federated methods train across decentralized data without sharing local data; representative approaches use global prototypes, encrypted prototype information, or feature enhancement.FedCMR aggregates client models at a central server to update a global model.Source: 7) Federated cross-modal retrieval · p. 18

8) Composed cross-modal retrieval:

  • Composed retrieval uses a reference image and modification text to retrieve target images; CMJPA treats modification text as an implicit transformation and aligns the inputs with the target.TFUN models three-input associations; AlRet uses composition and decomposition networks.Source: 8) Composed cross-modal retrieval · p. 18

9) Hierarchical/fine-grained cross-modal retrieval:

  • Hierarchical methods use hierarchical labels for multi-grained retrieval, while FGCrossNet provides a unified model for fine-grained multi-modal contexts.HiCH-Net combines hierarchical semantic learning with regularized cross-modal hashing.Source: 9) Hierarchical/fine-grained… · p. 18

IV. CROSS-MODAL RETRIEVAL BEYOND TEXT-IMAGE

Beyond text-image retrieval, the review applies a consistent taxonomy to text-video, text-audio, and other modality combinations, highlighting their distinct challenges and methods.

A. Text-video Cross-modal Retrieval

  • Text-video retrieval must align text with evolving video frames and specific moments; the review classifies work mainly as unsupervised real-value retrieval, with fewer supervised moment-retrieval and hashing studies.The methods are grouped as concept, CNN-RNN, GNN, Transformer, and vision-language pretraining approaches.Source: A. Text-video Cross-modal Retrieval · p. 19

1) Concept methods:

  • Concept methods use semantic concepts to match texts and videos, but limited selected concepts make rich temporal information difficult to capture.Earlier work uses classifiers for video objects and scenes and linguistic rules to extract text concepts.Source: 1) Concept methods · p. 19

2) CNN-RNN methods:

  • CNN-RNN methods encode text and video separately into a shared space; mean or max pooling aggregates frame sequences efficiently but trades precision for efficiency.Later methods add audio and motion cues, while CSL combines complementary latent and symbolic spaces.Source: 2) CNN-RNN methods · p. 19

3) GNN methods:

  • Text-video graph methods model component relationships with hierarchical or spatiotemporal graphs to capture global-to-local alignment and object interactions over time.HGR and GHAN use hierarchical graphs; STGs and GVSI model object interactions across time.Source: 3) GNN methods · p. 20

4) Transformer methods:

  • Transformer methods jointly encode video and text with mutual attention; MMT incorporates temporal information, while HiT uses hierarchical cross-modal contrastive matching.These methods also encode video audio and motion.Source: 4) Transformer methods · p. 20

5) VLP model methods:

  • Vision-language pretraining methods transfer text-image knowledge, primarily from CLIP, to text-video retrieval and incorporate temporal dependencies.VoP tunes cooperative prompts with fewer trainable parameters; Cap4Video integrates auxiliary captions, and T-MASS models concise text stochastically.Source: 5) VLP model methods · p. 20

B. Text-audio Cross-modal Retrieval

  • Text-audio retrieval aligns textual semantics with temporal audio sequences, and recent language-based work uses audio-caption datasets and is categorized as unsupervised real-value retrieval.The reviewed approaches include CNN-RNN and Transformer methods.Source: B. Text-audio Cross-modal Retrieval · p. 20

1) CNN-RNN methods:

  • CNN-RNN audio methods encode text and audio separately; MRL extracts localized audio features and adapts aggregation to varying granularity in text-audio associations.ATR uses NetRVLAD pooling, while OML [287] uses metric learning to align embeddings.Source: 1) CNN-RNN methods · p. 20

C. Image-audio Cross-modal Retrieval

The review surveys retrieval across image-audio, video-audio, image-3D, and other modality pairs, describing their alignment challenges and representative methods.

C. Image-audio Cross-modal Retrieval

  • Image-audio retrieval must bridge static visual features and dynamic audio; reviewed approaches span unsupervised and supervised real-value methods and supervised hashing.Examples use CNN-RNN and Transformer architectures, with alignment, metric learning, and hashing strategies.Source: C. Image-audio Cross-modal Retrieval · pp. 20–21

D. Video-audio Cross-modal Retrieval

  • Video-audio retrieval must model audio-video interactions over time; reviewed methods use sequential decisions, co-attention, attribute-guided interaction, or common subspaces.The review describes both unsupervised and supervised real-value methods, including Transformer and CNN-RNN approaches.Source: D. Video-audio Cross-modal Retrieval · p. 21

E. Image-3D Cross-modal Retrieval

  • Image-3D retrieval aligns 2D images with 3D shapes, predominantly point clouds, using supervised real-value methods such as metric learning and contrastive techniques.Reviewed approaches address modality discrepancy, noisy labels, and label scarcity through center losses, contrastive learning, prototypes, and hierarchical alignment.Source: E. Image-3D Cross-modal Retrieval · pp. 21–22

F. Broader Scope of Cross-modal Retrieval

1) Text-3D retrieval:

  • Text-3D retrieval aligns abstract language with spatial structures; xM Match uses BEV embeddings and cross-modal consistency learning.Its visual transformer models long-range dependencies.Source: 1) Text-3D retrieval · p. 22

2) Text-motion retrieval:

  • Text-motion retrieval matches descriptions to motion sequences, and TMR combines contrastive training with the motion-generation loss.The contrastive loss structures the cross-modal latent space.Source: 2) Text-motion retrieval · p. 22

3) Cross-modal 3D retrieval:

  • Cross-modal 3D retrieval addresses voxels, meshes, and point clouds; HSR combines global-to-global and local-to-local similarity with compact-set features.Joint losses optimize hierarchical similarity and cross-modal discrepancies.Source: 3) Cross-modal 3D retrieval · p. 22

4) Cross-modal image retrieval:

This section describes cross-modal image and image-video retrieval, then outlines the datasets and top-k metrics used to evaluate retrieval systems.

4) Cross-modal image retrieval:

  • Cross-modal image retrieval matches RGB images with depth, infrared, or sketches; SYRER models intra- and inter-class relations to address modality discrepancies.It uses a heterogeneous relationship contrast branch and point-wise depth extractor.Source: 4) Cross-modal image retrieval · p. 22

5) Image-video retrieval:

  • Image-video retrieval aligns static images with dynamic video sequences using temporal and spatial cues; LSVR directly compares image queries with videos.LSVR uses asymmetric comparison, Fisher vectors, and video descriptors rather than frame-based methods.Source: 5) Image-video retrieval · p. 22

A. Evaluation Dataset

  • The review compiles datasets for text-image retrieval and broader tasks including text-video, text-audio, image-audio, and image-3D retrieval.Examples include MS COCO and Flickr for text-image, MSR-VTT for video, Clotho for audio, and ModelNet40 for 3D.Source: A. Evaluation Dataset · pp. 22–23

B. Evaluation Metric

  • Common evaluation measures include Precision, Recall, mAP, and NDCG, with @k indicating that the metric considers the top-k retrieved results.These measures are calculated over retrieved results ranked within the specified cutoff.Source: B. Evaluation Metric · p. 23

1) Precision@k:

  • Precision@k averages, across queries, the proportion of top-k retrieved items that are relevant.The passage defines true positives and false positives as the relevant and non-relevant items among the top-k results.Source: 1) Precision@k · p. 23

2) Recall@k:

  • Recall@k averages, across queries, the proportion of all relevant items retrieved within the top k.The passages define TP and FN as true and false positives among the top-k results and identify Recall@k as a standard unsupervised cross-modal retrieval metric.Source: 2) Recall@k · p. 23
  • mAP@k averages each query’s top-k average precision, which is based on precision at relevant-result ranks.The passages define relevance indicators for retrieved results and specify averaging AP over the query set.Source: 3) mAP@k · p. 23

4) NDCG@k:

The review compares retrieval accuracy and hashing efficiency, finding performance patterns across supervision, model families, and hash lengths. It also surveys applications across search and content-generation settings.

4) NDCG@k:

  • NDCG@k discounts cumulative gain by rank and normalizes it against ideal gain, then averages scores across queries.The passages describe graded relevance and an ideal top-k set ordered by true correlation.Source: 4) NDCG@k · p. 23

C. Evaluation Result

  • Supervised methods generally achieve higher retrieval accuracy than unsupervised methods, though deep-learning advances are narrowing the gap.The review notes that deep models can be optimized without relying on expensive manual annotations.Source: C. Evaluation Result · p. 24
  • Deep retrieval models generally outperform shallow methods in accuracy, but BATCH can be comparable with much lower training and testing time cost.The review says deep networks’ representation capability has not been sufficiently exploited in existing methods.Source: C. Evaluation Result · p. 24
  • VLP methods achieve the highest unsupervised accuracy among reviewed real-valued methods, while supervised fine-tuned CLIP4CMR exceeds traditional supervised deep models.Transformer and VLP methods are reported to achieve superior retrieval accuracy to traditional deep networks.Source: C. Evaluation Result · p. 24
  • Hashing methods vary in how retrieval accuracy changes with longer codes, and additional bits do not always improve performance.The review notes that optimal hash-code length differs across methods.Source: C. Evaluation Result · p. 24
  • Shallow hashing methods are significantly more computationally efficient than deep hashing methods.The comparison covers training and testing time for 128-bit methods on a single Titan RTX GPU; among deep methods, architecture alone has no significant inherent efficiency advantage.Source: C. Evaluation Result · p. 25, Table XI · p. 25

VI. APPLICATION

  • Applications include product search, recipe search, scene-text retrieval, person re-identification, remote sensing, and medical-image search.These scenarios use text, voice, images, or descriptions as queries to retrieve relevant cross-modal content.Source: §VI · p. 25
  • Other applications include sketch-based image retrieval, video-moment retrieval, and retrieval-augmented generation.The review describes using sketches or text and visual cues to find relevant images or video moments, and retrieving information for content generation.Source: §VI · p. 25

VII. DISCUSSION AND OUTLOOK

The discussion identifies challenges in cross-modal retrieval and proposes research directions spanning efficiency, uncertain data, distribution change, model updating, user interaction, responsibility, and distributed systems. The conclusion reiterates the review’s taxonomy, methods, evaluation guidance, and outlook.

VII. DISCUSSION AND OUTLOOK

  • Efficient semantic modeling remains important because vision-language pretraining can capture complex semantics but requires substantial computation, storage, and energy.The review suggests brain-inspired processing and combining foundation models with human-generated knowledge as possible efficiency directions.Source: §VII · p. 26
  • Uncertain multimodal data—including imbalanced data, incomplete correspondences, noisy annotations, and misalignment—can impair retrieval models.Suggested approaches include probabilistic methods, self- or weakly supervised learning, and adversarial learning.Source: §VII · p. 26
  • Changes in modality distributions across time, location, and scenes can harm retrieval performance, motivating adaptation across domains.The review names domain adaptation, domain generalization, and adversarial learning as transfer approaches.Source: §VII · p. 26
  • Continuous updating is needed as database data is added or removed; online and adaptive learning can update models, while transfer and distillation can preserve learned representations.The review identifies avoiding repeated training from scratch and preventing catastrophic forgetting as relevant aims.Source: §VII · p. 26
  • User-centric retrieval can incorporate feedback and query modifications to adapt results to users’ changing intentions and preferences.The review also recommends interactive interfaces and visualizations for feedback, query changes, and result exploration.Source: §VII · pp. 26–27
  • Responsible retrieval requires attention to transparency, trustworthiness, fairness, and data bias, areas the review says remain insufficiently explored.It calls for explainability, robustness and privacy protections, fairness-aware models, diverse datasets, and debiasing algorithms.Source: §VII · p. 27
  • Distributed retrieval may reduce data transmission, accelerate retrieval, and improve privacy, but federated cross-modal retrieval remains at an early stage.The review discusses distributed computing, cloud infrastructure, and collaborative training across local sites.Source: §VII · p. 27

VIII. CONCLUSION

  • The paper concludes by presenting a taxonomy, reviewing methods and architectures, offering dataset and metric guidance, and outlining research opportunities and challenges.It frames these contributions as supporting understanding and development of cross-modal retrieval.Source: §VIII · p. 27

Related papers

Loading 2308.14263v3…