Source-linked AI summary
Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and Fusion
Yang Wang
TL;DR
Multi-modal analytics seeks to bridge semantic gaps and exploit complementary information across heterogeneous views. This survey reviews shallow and deep methods, emphasizing collaboration, adversarial competition, and fusion, and reports strong performance across benchmark tasks. It also identifies missing-view handling and other complexity and uncertainty issues as future challenges.
Problem
Multi-modal learning must bridge semantic gaps and capture correlations between heterogeneous, complementary modalities while handling challenges such as missing views.
Method
The paper provides a comprehensive survey of shallow and deep multi-modal methods, including collaborative and adversarial architectures, applications, and benchmark evaluations.
Results
Deep multi-modal methods show remarkable performance in fundamental problems, while surveyed benchmark results include competitive cross-modal retrieval performance.
Takeaways & Limitations
The survey identifies collaboration, adversarial competition, and fusion as critical components for combining complementary multi-modal information.
Takeaways & Limitations
The paper highlights timing constraints in spatial-temporal multi-modal collaboration: acting too early or too late limits the candidate's available information or ability to respond.
Abstract
from arXiv · showhide
With the development of web technology, multi-modal or multi-view data has surged as a major stream for big data, where each modal/view encodes individual property of data objects. Often, different modalities are complementary to each other. Such fact motivated a lot of research attention on fusing the multi-modal feature spaces to comprehensively characterize the data objects. Most of the existing state-of-the-art focused on how to fuse the energy or information from multi-modal spaces to deliver a superior performance over their counterparts with single modal. Recently, deep neural networks have exhibited as a powerful architecture to well capture the nonlinear distribution of high-dimensional multimedia data, so naturally does for multi-modal data. Substantial empirical studies are carried out to demonstrate its advantages that are benefited from deep multi-modal methods, which can essentially deepen the fusion from multi-modal deep feature spaces. In this paper, we provide a substantial overview of the existing state-of-the-arts on the filed of multi-modal data analytics from shallow to deep spaces. Throughout this survey, we further indicate that the critical components for this field go to collaboration, adversarial competition and fusion over multi-modal spaces. Finally, we share our viewpoints regarding some future directions on this field.
1 INTRODUCTION
Multi-modal data analytics addresses heterogeneous but complementary feature spaces by combining information across views. The introduction frames fusion strategies and deep architectures as ways to improve over single-view approaches.
- Multi-modal objects contain heterogeneous feature spaces, with each modality encoding a distinct property while complementing the others.
- Existing research combines information across multi-modal spaces to characterize objects and outperform single-view counterparts.
- Fusion strategies are broadly classified as early fusion, late fusion, and collaborative fusion.
- Early fusion processes combined multi-view information, whereas late fusion runs view-specific algorithms before combining their outputs.
- Collaborative fusion promotes interaction among modalities to achieve a shared consensus, distinguishing it from early and late fusion.
- Deep multi-modal architectures use multiple deep-network branches, one per modality, to model nonlinear data distributions and improve on shallow models.
2 DEEP MULTI-VIEW/MODAL LEARNING
Deep multi-view/modal learning uses deep networks to extract nonlinear high-level representations for clustering, classification, retrieval, recognition, and other applications. The surveyed methods combine information across views through architectures and objectives tailored to each task, including incomplete-view settings.
- Deep Multi-modal methods for Clustering and Classification: Deep networks extract high-level nonlinear representations that support multi-view clustering and classification.The survey reports remarkable performance in these fundamental problems and summarizes representative algorithms.
- Deep Multi-modal methods for Clustering and Classification: LMSC integrates multiple views into a latent representation, while MvSCN uses deep metric learning to model local view invariance.These methods target complementary aspects of multi-view structure: shared latent information and local invariance.
- Deep Multi-view Learning for Other Multimedia Applications: End-to-end methods incorporate view relationships through correlation regularization, attentive learning, and architectures that jointly exploit samples despite missing views.CorrReg concatenates individual-view features at an intermediate layer, while DEAML and CPM-Nets address recommendation and incomplete-view learning.
- Deep Multi-view Learning for Other Multimedia Applications: Deep multi-view learning also supports applications such as taxi-demand prediction, learning to rank, and biomedical medical concept normalization.DMVST-Net jointly models temporal, spatial, and semantic views, while MvSL2R learns joint and individual ranking objectives.
- Deep Multi-view Learning for Multimedia Analytics: Deep multi-view methods address multimedia retrieval, representation, and recognition through hashing, nonlinear distance metrics, autoencoders, CNNs, and multimodal feature fusion.Examples include compact hash codes, Deep-MDML for image retrieval, SkeletonNet, BCDA, MM-DFR, CDL, and MMCNN-MIML.
- Deep Generative Adversarial Networks for Multi-View Applications: GAN-based multi-view methods address missing-view imputation by using generators and discriminators to learn conditional or domain-to-domain mappings.Multi-view BiGAN, biconditional GAN, and VIGAN are presented as approaches for handling missing views.
3 DEEP CROSS-MODAL LEARNING
Deep cross-modal learning addresses semantic gaps between heterogeneous modalities through real-valued representation learning, hashing, GANs, transfer learning, and related architectures. Surveyed results use benchmark retrieval tasks to compare these approaches across datasets and query directions.
- Deep cross-modal learning seeks to bridge the semantic gap and capture correlations between heterogeneous modalities.
- Cross-Modal Representation and Hashing Learning: Real-valued methods learn common subspaces where cross-modal semantic similarity can be measured, while hashing methods learn representations in cross-modal Hamming space.Real-valued approaches include Deep-CCA, Corr-AE, CCL, CMDN, GIN, and GXN; hashing approaches include supervised and unsupervised models.
- Cross-Modal Real-Valued Representation Learning: Table 5 compares mAP for image-to-text, text-to-image, and average retrieval across Wikipedia, NUS-WIDE, and Pascal Sentence.Wikipedia generally yields lower precision because its categories are more abstract and semantically confusable than the concrete categories in NUS-WIDE and Pascal Sentence.
- Cross-Modal Hashing Learning: Table 6 and Table 7 compare deep cross-modal hashing methods for image-to-text and text-to-image retrieval on IAPRTC-12, MIRFlickr-25k, and NUS-WIDE using 16-, 32-, and 64-bit codes.The reported benchmarks include supervised and unsupervised hashing models.
- Cross-Modal Hashing Learning: For image queries, CM-DVStH achieves mAP = 0.882, 0.916 and 0.928, while CDQ shows competitive performance for text queries on MIRFlickr-25k and NUS-WIDE.MIRFlickr-25k contains 25,000 images across 24 relatively independent categories with discriminative features.
- GAN-Based and Other Deep Cross-Modal Methods: GAN-based methods learn modality-invariant or joint representations through adversarial objectives, while other approaches use transfer learning, correlation graphs, or supervision transfer.Examples include ACMR, CM-GANs, UCH, TANSS, MHTN, and related cross-modal frameworks.
4 POSSIBLE FUTURE DIRECTION
The survey proposes future multi-modal research that emphasizes collaboration between modalities rather than only fusing their information. A central challenge is determining where and when one modality should guide another under complexity and uncertainty.
- Future work should give greater weight to solving each modality’s problem by referring to information from other modalities.The proposed direction shifts attention toward inter-modal assistance in addition to information fusion.
- Spatial-Temporal Multi-modal Collaboration: A road-network example frames two modalities as candidates choosing paths toward destinations while sharing information to avoid traps.The example highlights spatial-temporal multi-modal collaboration within the same network.
- Spatial-Temporal Multi-modal Collaboration: The collaboration problem requires deciding where and when modalities should help each other.One candidate must identify the appropriate time slot to guide the other candidate in capturing information.
- Spatial-Temporal Multi-modal Collaboration: If guidance arrives too late, a candidate cannot step back; if it arrives too early, the candidate may not yet recognize the situation.
- Spatial-Temporal Multi-modal Collaboration: Ideal deep multi-modal models should learn powerful feature representations and spaces robustly enough to address these collaboration challenges in practical contexts.
5 CONCLUSIONS
The survey identifies multimodal collaboration, including adversarial collaboration, as central to fusing complementary information. It reviews state-of-the-art architectures, benchmark results, practical applications, and future directions addressing complexity and uncertainty.
- The survey identifies multimodal collaboration, including adversarial collaboration, as a critical issue for fusing complementary information into final outputs.
- It summarizes state-of-the-art deep multimodal and cross-modal architectures using benchmark multimodal datasets and reviews their practical applications.
- The proposed future directions address expensive complexity and uncertainty in multimodal data analytics.