PaperlayerView exact revision on arXiv

Source-linked AI summary

An Overview of Cross-media Retrieval: Concepts, Methodologies, Benchmarks and Challenges

Yuxin Peng, Xin Huang, Yunzhen Zhao

arXiv:1704.02223v4cs.MM

TL;DR

Cross-media retrieval addresses the media gap that makes heterogeneous media difficult to compare and retrieve across types. This paper reviews more than 100 references, organizes methodologies and challenges, and builds benchmarks including the XMedia dataset. It reports representative experimental results across multi-modality and bi-modality settings and identifies continuing opportunities for improved accuracy, efficiency, and deep-neural-network applications.

  • Problem

    Cross-media retrieval must address inconsistent representations across media types, while existing concepts, methodologies, and benchmarks remain unclear in the literature.

  • Method

    The paper reviews more than 100 references, categorizes cross-media retrieval methods, builds benchmarks with datasets and experimental results, and constructs XMedia.

  • Results

    S2UPG achieves the best reported results in both multi-modality and bi-modality experiments, while the overview provides datasets and experimental benchmarks for evaluation.

  • Takeaways & Limitations

    The benchmarks can be directly adopted for prompt evaluation, allowing researchers to focus on algorithm design rather than repeatedly comparing methods and results.

Abstract

from arXiv · show

Multimedia retrieval plays an indispensable role in big data utilization. Past efforts mainly focused on single-media retrieval. However, the requirements of users are highly flexible, such as retrieving the relevant audio clips with one query of image. So challenges stemming from the "media gap", which means that representations of different media types are inconsistent, have attracted increasing attention. Cross-media retrieval is designed for the scenarios where the queries and retrieval results are of different media types. As a relatively new research topic, its concepts, methodologies and benchmarks are still not clear in the literatures. To address these issues, we review more than 100 references, give an overview including the concepts, methodologies, major challenges and open issues, as well as build up the benchmarks including datasets and experimental results. Researchers can directly adopt the benchmarks to promptly evaluate their proposed methods. This will help them to focus on algorithm design, rather than the time-consuming compared methods and results. It is noted that we have constructed a new dataset XMedia, which is the first publicly available dataset with up to five media types (text, image, video, audio and 3D model). We believe this overview will attract more researchers to focus on cross-media retrieval and be helpful to them.

I. Introduction

Cross-media retrieval addresses flexible queries across heterogeneous media, where inconsistent representations create a media gap. The paper reviews existing work, builds evaluation benchmarks, and introduces the five-media XMedia dataset.

  • Motivation: Cross-media retrieval lets users submit one media type as a query and retrieve relevant results in other media types.Examples include retrieving text, images, videos, audio clips, and 3D models from a photograph.
  • Challenges: The central challenge is the media gap: different media representations occupy inconsistent feature spaces, making cross-media similarity difficult to measure.Existing methods mainly address this through common spaces or direct cross-media similarity measurement.
  • Contributions: The overview reviews more than 100 references to clarify cross-media retrieval concepts, methodologies, benchmarks, challenges, and open issues.It summarizes existing works and methodologies to facilitate further research.
  • Contributions: The paper builds benchmarks with datasets and experimental results that researchers can directly adopt for prompt evaluation.This is intended to reduce time spent comparing methods and results.
  • Contributions: XMedia is introduced as the first publicly available dataset containing up to five media types: text, image, video, audio, and 3D model.It is designed for comprehensive evaluations of cross-media retrieval.

II. Definition of Cross-media Retrieval

Cross-media retrieval retrieves relevant instances of different media types for a query in any media type by computing cross-media similarities. Its formulation uses paired heterogeneous training data, testing data, semantic labels, and supervision settings.

  • Formulation: Training data consists of paired instances from two media types, X and Y, whose co-existence relationships describe shared relevant semantics.Semantic category labels may also be provided for the training instances.
  • Formulation: Testing data contains instances from both media types, and the retrieval goal is to compute sim(x_a, y_b) for cross-media retrieval.The system retrieves relevant instances of different media types for a query of either type.
  • Supervision settings: Unsupervised, semi-supervised, and fully supervised methods differ according to whether none, some, or all training data are labeled.Relevance-feedback methods are separately identified when they cannot be easily categorized by supervision setting.
  • Scope: Cross-media retrieval differs from image annotation and captioning because it retrieves existing descriptions or media instances rather than generating tags or descriptions.It is also distinguished from transfer learning as a specific retrieval task focused on the media gap.

III. Common Space Learning

Common space learning is a mainstream approach that maps heterogeneous media into a shared space where semantically related instances can be compared. The overview organizes these methods by model, correlation modeling, and common-space properties.

  • Core idea: Common space learning assumes that media sharing semantics have latent correlations and can be brought close in a common high-level semantic space.For example, multiple media descriptions of the Golden Gate Bridge may occupy nearby positions.
  • Taxonomy: The overview divides existing common space learning methods into seven categories spanning statistical correlation analysis, DNNs, graphs, metrics, ranking, dictionaries, and hashing.The categories overlap because they are classified according to different aspects.
  • Method categories: DNN-based methods use deep neural networks to learn cross-media correlations, while graph, metric-learning, and ranking methods encode relationships through their respective objectives.Dictionary learning uses sparse coefficients, and cross-media hashing learns a common Hamming space to accelerate retrieval.
  • Traditional statistical correlation analysis: Traditional statistical correlation analysis learns linear projection matrices by optimizing statistical values and forms a basic paradigm for common space learning.CCA is a representative method that maximizes pairwise correlations between heterogeneous data sets.
  • Traditional statistical correlation analysis: Semantic extensions of CCA incorporate category labels or additional semantic views, and the reported results indicate that semantic information improves retrieval accuracy.Examples include supervised, multi-view, and multi-label CCA variants.

B. DNN-based Methods

DNN-based methods learn nonlinear common representations for heterogeneous media, using either shared layers or subnetworks coupled by correlation constraints. Their abstraction power is offset by dependence on large labeled datasets and the difficulty of handling more than two media types.

  • B. DNN-based Methods: DNNs learn nonlinear relationships for common-space learning across different media types.Bimodal deep autoencoders and related architectures use neural networks to transform heterogeneous inputs.
  • B. DNN-based Methods: DCCA extends CCA nonlinearly by using separate subnetworks whose code-layer correlation constraints maximize total correlation.This architecture differs from shared-layer networks by coupling two subnetworks at their code layers.
  • B. DNN-based Methods: Deep architectures use either a shared layer for different media or subnetworks coupled by correlation constraints at code layers.Both designs use DNNs to abstract complex cross-media correlations.
  • B. DNN-based Methods: Large-scale labeled cross-media datasets are difficult to collect, limiting the training conditions for DNN models.The paper notes that most existing works also handle only two input media types, although some recent methods use more than two.
  • B. DNN-based Methods: RNNs and LSTMs have been applied to image or video captioning to generate textual descriptions of visual content.These architectures are presented as related deep methods for multimedia applications.
  • B. DNN-based Methods: GANs estimate generative models through adversarial training of a generative model and a discriminative model.The two models are trained simultaneously in an adversarial process.

C. Cross-media Graph Regularization Methods

Graph regularization methods learn cross-media representations by modeling semantic and similarity relationships with graphs. They can integrate multiple media types and correlation sources, but graph construction is computationally expensive and metric-learning variants require supervision.

  • C. Cross-media Graph Regularization Methods: Graph regularization enriches training data and supports joint projection learning by exploiting affinities among labeled and unlabeled instances.JRL constructs a separate graph for each media type, while later work uses a unified hypergraph across media types.
  • C. Cross-media Graph Regularization Methods: JRL jointly considers correlations and semantic information for up to five media types through separate media-specific graphs.Unified hypergraph methods allow different media types to reinforce one another and can exploit instance-level segmentation.
  • C. Cross-media Graph Regularization Methods: Graph regularization represents semantic relevance, intra-media similarities, and inter-media similarities within cross-media correlation learning.These multiple correlation types make it suitable for unified modeling beyond two media types.
  • C. Cross-media Graph Regularization Methods: Metric learning transforms input features so similar instances become close and dissimilar instances become separated.Cross-media metric learning preserves semantic similar/dissimilar information during common-space learning.
  • C. Cross-media Graph Regularization Methods: Existing cross-media metric-learning methods depend on supervision information and are inapplicable when supervision is unavailable.JGRHML additionally uses joint graph regularization to combine complementary media information.
  • C. Cross-media Graph Regularization Methods: Dictionary learning decomposes data into media-specific dictionaries and common sparse coefficients or mappings across modalities.Sparse coefficients provide the common representation for modeling cross-media correlations.
  • C. Cross-media Graph Regularization Methods: Dictionary-learning optimization becomes challenging on large-scale cross-media data because it requires extensive calculation.Its sparsity can make cross-media correlations easier to capture, but does not remove the computational burden.

G. Cross-media Hashing Methods

Cross-media hashing maps multiple media types into a common Hamming space to accelerate retrieval with short binary codes. Methods incorporate cross-media, intra-media, and semantic information, but evaluations often use only two media types and datasets too small to validate efficiency.

  • G. Cross-media Hashing Methods: Cross-media hashing generates hash codes for multiple media types and projects their data into a common Hamming space.It extends hashing beyond methods designed for a single media type.
  • G. Cross-media Hashing Methods: Some methods jointly learn common-space projections and composite quantizers instead of separating continuous learning from binary-code generation.This seamless scheme contrasts with the two-stage design used by most existing works.
  • G. Cross-media Hashing Methods: Hashing accelerates retrieval by using short binary codes, making it useful for large-scale datasets.The paper motivates hashing by the dramatic growth of multimedia data and retrieval-efficiency requirements.
  • G. Cross-media Hashing Methods: Cross-media hashing methods preserve inter-media similarities, intra-media similarities, and semantic discriminative capability.These methods learn from cross-media correlations to bridge the media gap.
  • G. Cross-media Hashing Methods: Most recent cross-media hashing works retrieve between only two media types, mainly image and text.The paper notes that small datasets such as Wikipedia and Pascal Sentence are commonly used for evaluation.
  • G. Cross-media Hashing Methods: Small evaluation datasets prevent effective validation of hashing’s efficiency advantage.Examples include Wikipedia with 2,866 image/text pairs and Pascal Sentence with 1,000 image/text pairs.
  • G. Cross-media Hashing Methods: Other common-space methods include two-step projections, local projections, task-specific projections, and manifold alignment.These approaches address limitations of single global projections for complex data or multiple retrieval tasks.

IV. Cross-media Similarity Measurement

Cross-media similarity measurement avoids explicitly projecting heterogeneous instances into a common space by using dataset relationships, graphs, or nearest-neighbor analysis. These methods can bridge out-of-dataset queries, but graph construction, noisy relationships, and unstable neighbors limit practical reliability.

  • IV. Cross-media Similarity Measurement: Direct similarity methods use known media instances and dataset correlations to bridge the media gap without explicit common-space projection.Because no common space exists, ordinary distance measures and classifiers cannot directly compute cross-media similarities.
  • IV. Cross-media Similarity Measurement: Graph-based methods represent media instances or multimedia documents as vertices connected by cross-media correlations.Graphs may jointly use single-media similarities, co-existence relationships, and semantic labels.
  • IV. Cross-media Similarity Measurement: Graph-based approaches differ in whether they construct separate graphs for each medium or one uniform graph integrating all media types.Other strategies combine media-specific and multimedia-document graphs for semantic propagation.
  • IV. Cross-media Similarity Measurement: Graph-based methods are time- and space-consuming because they require graph construction.Their effectiveness also commonly depends on available multimedia-document relevance or relevance feedback.
  • IV. Cross-media Similarity Measurement: Unavailable, noisy, or incomplete multimedia-document relationships make graph-based cross-media retrieval difficult in real-world applications.Queries outside the datasets are especially difficult when the required relevance relationships are unavailable.
  • IV. Cross-media Similarity Measurement: Neighbor analysis retrieves nearest neighbors using single-media similarities and uses their associated content as cross-media results or expanded queries.This mechanism can provide bridges for queries that are outside the dataset.
  • IV. Cross-media Similarity Measurement: Neighbor analysis has high time and space complexity and may produce unstable performance because neighbor relevance is difficult to ensure.Its flexibility does not eliminate the computational issues associated with graph construction.

V. Other Methods for Cross-media Retrieval

The paper surveys relevance feedback and multimodal topic models as complementary approaches for cross-media retrieval beyond common-space learning and similarity measurement.

  • Multimodal Topic Model: Multimodal topic models extend text-processing topic models to represent cross-media data at the topic level.Cross-media similarities are commonly obtained through conditional probabilities.
  • Relevance Feedback Analysis: Relevance feedback uses user-provided information to improve cross-media retrieval accuracy and bridge the media gap.Short-term feedback uses the current user's labels, whereas long-term feedback aggregates information from all users.
  • Relevance Feedback Analysis: Short-term relevance feedback refines similarities from user-labeled positive and negative nearest neighbors when queries fall outside the datasets.
  • Relevance Feedback Analysis: Long-term relevance feedback converts global feedback into pairwise similar/dissimilar constraints or refines a shared vector representation.Some methods combine long-term and short-term feedback by modeling global feedback structure before refining representations.
  • Multimodal Topic Model: Topic-model approaches include Corr-LDA, tr-mmLDA, MDRF, and supervised topic models, with tr-mmLDA correlating separate image and text topic sets through regression.tr-mmLDA assumes each text word has a visual interpretation, while MDRF is formulated as a Markov random field over LDA topic models.

VI. Cross-Media Retrieval Dataset

The paper reviews widely used cross-media datasets, introduces XMedia for five-media evaluation, and describes feature representations and dataset splits used in experiments.

  • Dataset Overview: Wikipedia and NUS-WIDE are the most widely used cross-media retrieval datasets, while Pascal VOC and Clickture provide additional evaluation resources.The review summarizes dataset usage frequencies and describes datasets designed for different cross-media retrieval settings.
  • Wikipedia Dataset: Wikipedia contains 2,866 image/text pairs drawn from 10 populated categories of featured Wikipedia articles.
  • XMedia Dataset: XMedia covers 20 categories and five media types: text, image, video, audio, and 3D model.It contains 12,000 publicly available media instances and was constructed for comprehensive and fair evaluation.
  • NUS-WIDE Dataset: NUS-WIDE contains 269,648 images across 81 concepts and 425,059 associated tags after duplicate removal.Low-frequency tags absent from WordNet are removed to improve tag quality.
  • Clickture Dataset: Clickture is a large-scale click-based dataset containing 40 million images and 73.6 million text queries, with Clickture-Lite used for evaluation.Clickture-Lite includes 1.0 million images, 11.7 million queries, and 23.1 million query-image-click training triads.
  • Feature Extraction and Dataset Split: Experiments represent text with 10-topic LDA or BoW features, images with BoVW or CNN features, audio with MFCCs, and 3D models with LightField descriptors.
  • Feature Extraction and Dataset Split: XMedia uses a 4:1 training/testing ratio, yielding 9,600 training instances and 2,400 testing instances.

B. Evaluation Metrics and Compared Methods

The evaluation covers multi-modality and bi-modality retrieval using MAP and additional measures, showing that methods exploiting richer cross-media structure generally perform best.

  • Evaluation Tasks: Multi-modality retrieval submits any media type as a query and retrieves all media types, whereas bi-modality retrieval retrieves the other media type.
  • Evaluation Metrics: MAP scores evaluate retrieval on most datasets, supplemented by precision-recall curves and running time for broader assessment.Clickture instead uses relevance information from text-query and image interactions because it lacks category labels for MAP and PR evaluation.
  • Experimental Results: S2UPG achieves the best multi-modality retrieval results by modeling fine-grained media patches in a unified hypergraph.Methods incorporating semantic information generally outperform methods based only on pairwise correlations.
  • Experimental Results: S2UPG also achieves the best bi-modality results, while CMCP, HSNN, JGRHML, and JRL generally outperform BITR, CCA, and CCA+SMN.The relative trends differ between bi-modality and multi-modality retrieval, with several methods close on bi-modality tasks.
  • Experimental Results: Clickture results show overall method trends similar to those observed on the other datasets.
  • Feature Comparison: Using CNN image/video features and BoW text features, the paper reports average MAP scores for Wikipedia retrieval tasks.Detailed results for other datasets and tasks are provided on the project website.

VIII. Challenges and Open Issues

The paper identifies dataset quality, accuracy and efficiency, deep neural networks, context correlations, and practical deployment as central open issues in cross-media retrieval.

  • Dataset Construction and Benchmark Standardization: Existing cross-media datasets remain limited in size, media-type coverage, and category design, constraining objective evaluation.The paper highlights semantic overlap and confusion in Wikipedia’s high-level categories as one evaluation problem.
  • Dataset Construction and Benchmark Standardization: XMediaNet is being constructed with five media types, 200 WordNet categories, and more than 100,000 media instances for broader evaluation.The planned categories include 48 animal classes and 152 artifact classes, with data crawled from multiple websites.
  • Dataset Construction and Benchmark Standardization: Continuously updated benchmarks can let researchers evaluate methods directly instead of repeatedly comparing methods and results.The paper presents benchmark datasets, experimental results, and an updating website as infrastructure for research evaluation.
  • Improvement of Accuracy and Efficiency: Cross-media retrieval still requires methods that improve accuracy while remaining efficient, including richer context for graph construction and stronger single-media features.The paper identifies link relationships as one possible source of additional context information.
  • Applications of Deep Neural Network: Deep neural network research should move beyond feature-dependent common-space extensions toward end-to-end architectures using original media instances.The paper also notes that many existing methods handle only two media types and suggests incorporating modality-specific networks into unified frameworks.
  • Exploitation of Context Correlation Information: Context information remains underused even though cross-media correlations can arise from relationships such as links between web pages.The paper connects this limitation to inaccurate retrieval on real-world cross-media data.

IX. Conclusion

The paper surveys cross-media retrieval methods, datasets, benchmarks, challenges, and open issues. It also presents XMedia as the first five-media-type dataset and describes XMediaNet as a larger dataset under construction.

  • Cross-media retrieval addresses the media gap by enabling retrieval across different media types.
  • The overview reviews more than 100 references and organizes methods into common space learning and cross-media similarity measurement.
  • XMedia is presented as the first dataset with five media types for comprehensive and fair evaluation, while XMediaNet is being constructed with more than 100,000 instances.
  • The paper provides benchmark datasets, compared methods, evaluation metrics, experimental results, and a continuously updated website.
Loading 1704.02223v4…