PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review on Methods and Applications in Multimodal Deep Learning

Jabeen Summaira, Xi Li, Amin Muhammad Shoib, Jabbar Abdul

arXiv:2202.09195v1cs.LGcs.MM

TL;DR

Unimodal learning does not cover all aspects of human learning, motivating multimodal deep learning that links information across modalities. This survey reviews MMDL applications and methods across seven modalities, proposes a fine-grained taxonomy, compares architectures and evaluations, and discusses domain-specific issues and future directions.

  • Problem

    Unimodal learning cannot cover all aspects of human learning, while heterogeneous modalities can provide contextually related and supplementary information.

  • Method

    The article reviews recent MMDL models and applications across image, video, text, audio, body gesture, facial expression, and physiological signal modalities, organizing methods into a fine-grained taxonomy.

  • Results

    The survey provides comparative analyses of MMDL application groups, including model architectures, modalities, datasets, features, and evaluation metrics.

  • Takeaways & Limitations

    MMDL research spans diverse applications and technical challenges, with the review identifying domain-specific issues and possible future research directions.

  • Takeaways & Limitations

    Automated emotion systems remain limited by restricted datasets, motivating multimodal recordings with more representative subject collections.

Abstract

from arXiv · show

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning (MMDL) is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This paper focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, and physiological signals. Detailed analysis of the baseline approaches and an in-depth study of recent advancements during the last five years (2017 to 2021) in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning methods is proposed, elaborating on different applications in more depth. Lastly, main issues are highlighted separately for each domain, along with their possible future research directions.

1 Introduction:

Multimodal deep learning links information across heterogeneous modalities to support richer understanding and predictions. This survey organizes recent methods, applications, comparative evidence, challenges, and future directions across a broader modality set than earlier reviews.

  • Combining contextually related modalities can reveal complementary information unavailable from individual modalities and support more appropriate predictions.
  • MMDL processes and relates information from image, video, text, audio, body gesture, facial expression, and physiological-signal modalities.
  • The paper proposes a fine-grained taxonomy and analyzes architectures, feature extractors, datasets, evaluation metrics, and model features across MMDL applications.
  • Earlier surveys commonly emphasized image, video, text, and audio, whereas this review additionally covers body gestures, facial expressions, and physiological signals.
  • It compares MMDL models on benchmark datasets and metrics while highlighting major challenges, open problems, and possible future research directions.

2 Background:

Multimodal learning has developed through successive historical eras and now faces challenges arising from heterogeneous, complementary, and sometimes incomplete data. The survey focuses on the deep-learning era and frames these challenges around representation, translation, and broader multimodal processing.

  • Multimodal machine learning supports applications including audiovisual speech recognition, retrieval, emotion recognition, captioning, VQA, and health analysis.
  • The paper concentrates mainly on the deep learning era within a four-era historical account of multimodal applications.
  • The survey supplements its main text with detailed architectures, datasets, and evaluation metrics in supplementary sections.
  • 2.1 Challenges of multimodal learning:: Heterogeneous multimodal data create challenges because sources may contain redundant, complementary, noisy, or missing information.
  • 2.1 Challenges of multimodal learning:: Multimodal representation encodes information from several media into tensor or vector forms using joint and coordinated representations.
  • 2.1 Challenges of multimodal learning:: Multimodal translation maps one modality to another, but translation quality is difficult to assess because images, speech, and descriptions may lack a single correct output.

3 MMDL Methods and Applications:

The survey groups multimodal deep learning methods and applications by their relevance and dominance across research areas. It introduces image description as one application combining computer vision and natural language processing.

  • MMDL methods and applications are grouped into related categories spanning multiple research areas and summarized in a taxonomy.
  • Image description generates textual descriptions from input images by combining computer vision and natural language processing.
  • Image description frameworks are categorized as retrieval-based, template-based, or deep-learning-based approaches.

3.1 Multimodal Image Description

Multimodal image description methods use visual information to generate text through encoder-decoder, semantic-concept, and attention-based designs. The surveyed literature also extends image description toward stylistic, guided, unsupervised, and context-sensitive caption generation.

  • Encoder-decoder-based image description: EDID models typically use CNN encoders to extract image features and RNN decoders to generate captions.
  • Captioning research includes romantic, negative, positive, factual, and humorous styles, alongside PoS-guided and unsupervised frameworks.
  • Semantic concept-based image description: SCID methods selectively extract semantic concepts, merge them with image features and language-model hidden states, and generate concept-based descriptions.
  • Attention-based image description: AID methods focus on distinct image regions according to context through attention mechanisms, including local and global attention.
  • Video description is treated separately as textual generation from visual video content using video and text modalities.

3.2 Multimodal Video Description

Multimodal video description combines video and text through encoder–decoder architectures, reinforcement learning, and comparative evaluation across models, datasets, and metrics.

  • The section presents general diagrams for CNN-RNN and RNN-RNN video description architectures and a comparative video-description model table.
  • Video description uses video and text modalities, with approaches comparatively analyzed by architecture, multimedia, datasets, and evaluation metrics.
  • CNN-RNN Architectures: CNN-RNN is the most broadly used architecture combination, using CNN variants for visual encoding and RNN variants for sentence decoding.
  • RNN-RNN Architectures: RNN-RNN video description uses RNN variants for both visual-content extraction and sentence generation.
  • Deep Reinforcement Learning Architectures: Deep reinforcement learning approaches reward or penalize actions toward target outcomes and include hierarchical, frame-selection, multitask, and event-sequence methods.

3.3 Multimodal Visual Question Answering (MMVQA):

Multimodal visual question answering combines visual inputs with natural-language questions and is organized around joint embeddings, attention, and external knowledge.

  • VQA processes extracted image or video features together with a question to answer natural-language questions about visual content.
  • VQA is more complex than related vision-language functions because questions are open-ended, visual information is high-dimensional, and several computer-vision subtasks are involved.
  • Multimodal Joint-Embedding Models: Multimodal joint-embedding models learn representations from multiple modalities in a common feature space and support cross-modal reasoning.
  • Multimodal Attention-based Models: Multimodal attention-based models prioritize local image or video features and can use co-attention, bilinear pooling, graph attention, or re-attention mechanisms.
  • The comparative analysis covers joint-embedding, attention-based, and external-knowledge-based VQA models and their evaluation metrics.
  • Multimodal External Knowledge-based Models: External-knowledge-based VQA models link questions and visual inputs with knowledge bases such as DBpedia, ConceptNet, WebChild, Freebase, and WordNet.

3.4 Multimodal Speech synthesis (MMSS):

Multimodal speech synthesis converts text into spoken audio using deep neural architectures, with research emphasizing naturalness and intelligibility.

  • Text-to-speech converts natural-language text into a spoken waveform in real time and is evaluated through naturalness, intelligibility, and listener preferences.
  • Deep-learning TTS uses DNN architectures to model relationships between text and acoustic realizations without human preprocessing.
  • The section compares deep-learning TTS approaches using text, audio, and video modalities and accuracy evaluation.
  • Tacotron uses a sequence-to-sequence encoder, attention-based decoder, and post-processing architecture to synthesize waveforms from text and audio pairs.
  • A VAE’s one-to-many mapping is reported to improve TTS performance and speech naturalness while supporting optimum synthesis on modern hardware.

3.5 Other MMDL Applications:

Other multimodal deep-learning applications include emotion recognition and event detection, using combinations of audiovisual, textual, facial, gestural, and physiological signals.

  • Multimodal Emotion Recognition: Multimodal emotion-analysis methods include facial-expression and EEG fusion, audio-video streams, and contextual or attention-based conversational frameworks.
  • Multimodal Emotion Recognition: Multimodal emotion recognition combines modalities including text, audio, video, physiological signals, facial expressions, and body gestures.
  • Multimodal Emotion Recognition: An empirical study reports that automatic emotion analysis can increase response accuracy and enable more rapid anticipation of a subject’s emotional state.
  • Multimodal Event Detection: Multimodal event-detection systems identify actions and events from combinations of images, videos, audio, and text in media streams.
  • Multimodal Event Detection: Event-detection research includes social-media event classification, unsupervised anomaly detection in crowded scenes, and unified image-text representations across social and news domains.
  • The comparative analysis covers multimodal emotion-recognition and event-detection models using modalities such as physiological signals, facial expressions, body gestures, images, video, audio, and text.

4 Experimental results comparisons on benchmark Datasets and Evaluation Metrics:

The survey compares multimodal models across benchmark datasets and evaluation metrics in image description, video description, VQA, and speech synthesis. Results identify leading models while showing that speech-quality assessment remains constrained by objective metrics.

  • 4.1 Multimodal Image Description Results:: GET achieves the best MS-COCO results among the compared multimodal image description models across standard evaluation metrics.Its architecture merges local and global features and fuses relevant decoder information through a global gated adaptive controller.
  • 4.1 Multimodal Image Description Results:: GET outperforms comparative image description models on B_1, B_4, M, and R, while VSR performs better on C and S.
  • 4.2 Multimodal Video Description Results:: SemSynAN achieves the best B, C, M, and R results on both MSR-VTT and MSVD among the compared video description models.It combines syntactic, visual, and semantic representations, using 2D-CNN and 3D-CNN visual feature extractors.
  • 4.3 Multimodal Visual Question Answering Results:: MCAN achieves the highest overall accuracy on both test-dev and test-std among the compared VQA models.Its multimodal co-attention layer uses image features and question self-attention to refine representations for reasoning.
  • 4.4 Multimodal Speech synthesis: MOS and MCD are commonly used subjective and objective speech-synthesis metrics, but objective measures cannot indicate produced speech quality.Subjective assessments require substantial resources and raise validity, reliability, and reproducibility concerns.

5 Discussion and Future Direction:

The discussion identifies domain-specific limitations across multimodal applications and outlines corresponding directions for improving data, models, feature extraction, and cross-modal integration.

  • Multimodal Image Description: Image captioning remains constrained by language-generation quality, training-data quality and quantity, domain-specific datasets, and limited exploration of unsupervised or reinforcement learning.Future directions include large-scale detailed datasets, open-source evaluation, advanced attention, and region-based captioning.
  • Multimodal Video Description: Video description remains below human performance, with future work emphasizing visual dialogue, audio-feature extraction, and broader real-world event coverage.Audio can provide information when visual clues are unavailable, such as sounds from water, animals, vehicles, or instruments.
  • Visual Question Answering: VQA needs more suitable visual feature extraction because uniform frame-box features do not track objects through semantic segmentation.The discussion also calls for goal-oriented VQA datasets.
  • Speech Synthesis: Speech synthesis is limited by the need for large quantities of high-quality text-speech pairs and by incomplete front-end text analysis.Future work targets data-efficient end-to-end training using large-scale unpaired text and speech recordings.
  • Multimodal Emotion Recognition: Emotion recognition requires more representative multimodal recordings and improved handling of physiological-signal preprocessing.The discussion specifically identifies ECG, EMG, and skin-temperature signals as emerging areas for further study.
  • Multimodal Event Detection: Multimodal event detection faces dimensionality growth from feature concatenation and needs joint textual, audiovisual, and cross-platform modeling.Suggested directions include improved feature learning and transfer learning across multiple social platforms.

6 Conclusion:

The survey reviews recent multimodal deep-learning advances across diverse modalities and application groups. It organizes methods through a fine-grained taxonomy, compares architectures and evaluations, and identifies domain-specific research directions.

  • Scope: The survey covers MMDL applications using image, audio, video, text, body gestures, facial expressions, and physiological signals.This scope is presented as broader than previous related surveys.
  • Taxonomy: A fine-grained taxonomy categorizes MMDL methods into different application groups.The taxonomy is accompanied by discussion of architectures, datasets, features, and evaluation metrics.
  • Comparative Analysis: Comparative analyses report experimental results for MMDL applications on benchmark datasets and evaluation metrics.The comparisons address model architectures, media, datasets, features, and metrics for each application group.
  • Future Directions: The survey lists open research problems and possible future directions separately for each application group.The authors expect these taxonomy and research-direction components to promote future MMDL studies.
Loading 2202.09195v1…