PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Machine Learning: A Survey and Taxonomy

Tadas Baltrušaitis, Chaitanya Ahuja, Louis-Philippe Morency

arXiv:1705.09406v2cs.LG

TL;DR

Multimodal machine learning must process and relate information across modalities to help AI interpret the world. This survey organizes recent advances into a taxonomy of five broader challenges, providing a framework for understanding the field and identifying future research directions.

  • Problem

    AI needs methods that interpret and reason about multimodal messages by processing and relating information across multiple modalities.

  • Method

    The paper surveys recent multimodal machine learning advances and organizes them beyond early-versus-late fusion into a common taxonomy.

  • Results

    The taxonomy identifies five challenges: representation, translation, fusion, alignment, and co-learning.

  • Takeaways & Limitations

    The taxonomy offers a framework for understanding multimodal machine learning and identifying directions for future research.

  • Takeaways & Limitations

    DBM-based multimodal representations have high computational training costs and require approximate variational training methods.

Abstract

from arXiv · show

Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.

1 INTRODUCTION

Multimodal machine learning seeks to help AI interpret and reason about messages spanning multiple modalities by building models that process and relate multimodal information. The field’s heterogeneity creates distinctive challenges, motivating a taxonomy organized around five core technical challenges beyond the usual early- versus late-fusion split.

  • Multimodal research problems and datasets involve multiple modalities, such as vision, touch, sound, smell, and other channels of experience.
  • Multimodal machine learning builds models that process and relate information from multiple modalities to advance AI’s understanding of the world.The field spans research from audio-visual speech recognition to language and vision models.
  • Heterogeneous multimodal data create unique computational challenges, while learning across modalities can capture correspondences and deepen understanding of natural phenomena.
  • The paper identifies five core technical challenges that must be addressed to advance multimodal machine learning.
  • The taxonomy goes beyond the typical early- and late-fusion split by structuring challenges and related sub-challenges in multimodal learning.Representation is described as learning summaries that exploit complementarity and redundancy despite modality heterogeneity.

2 APPLICATIONS: A HISTORICAL PERSPECTIVE

Multimodal applications evolved from audio-visual speech recognition to multimedia indexing, multimodal interaction, and language-and-vision media description. Across these applications, multimodal benefits depend on context, while evaluation and technical challenges remain central.

  • Historical progression: Multimodal applications span audio-visual speech recognition, image captioning, multimedia indexing and retrieval, and multimodal interaction.The paper presents this progression as a brief historical perspective on multimodal applications.
  • Audio-visual speech recognition: AVSR primarily improves robustness in noisy speech, because visual information helps at low signal-to-noise ratios but not in noiseless scenarios.The captured interactions were supplementary rather than complementary, since both modalities contained the same information.
  • Multimedia indexing and retrieval: Multimedia indexing and retrieval emerged as digitized multimedia expanded and researchers moved beyond keyword-based searches toward visual and multimodal content.The growth of personal computers and the internet increased the quantity of digitized multimedia content.
  • Multimodal interaction: Multimodal interaction research began in the early 2000s to understand human multimodal behavior during social interactions, supported by datasets such as the AMI Meeting Corpus.The AMI Meeting Corpus contains more than 100 hours of fully transcribed and annotated meeting videos.
  • Multimodal interaction: Multimodal affect recognition often improves with multiple modalities, but the improvement is reduced for naturally-occurring emotions.This finding comes from a meta-analysis of recent work.
  • Language and vision: Language-and-vision media description, especially image captioning, aims to generate text descriptions of images, but evaluating description quality remains a main challenge.Image captioning is motivated partly by helping visually impaired people with daily tasks.

3 MULTIMODAL REPRESENTATIONS

The paper organizes multimodal representations into joint and coordinated types, differing in whether modalities share one space or occupy separate aligned spaces. Joint representations suit settings with complete multimodal input at inference, whereas coordinated representations support separate modality processing but remain limited beyond two modalities.

  • Representation taxonomy: Multimodal representations are categorized as joint or coordinated: joint representations combine modalities in one space, while coordinated representations separately process modalities and enforce similarity constraints.Coordination can use cosine-distance minimization, correlation maximization, or partial-order constraints.
  • Joint representations: Neural joint representations often improve performance and can be pre-trained unsupervised, but their gains depend on training data and they do not naturally handle missing modalities.Multimodal autoencoders fuse individually encoded modalities into a multimodal representation.
  • Joint representations: Multimodal deep Boltzmann machines naturally handle missing data, including an absent modality, and can generate one modality from another, but they are difficult and costly to train.They can also generate both modalities from the learned representation and use unlabeled data through unsupervised training.
  • Joint representations: Joint representations project multimodal data into a common space and are best suited when all modalities are present during inference.They have been used for audio-visual speech recognition, affect, and multimodal gesture recognition.
  • Coordinated representations: Coordinated representations project each modality into separate but coordinated spaces, making them suitable when modalities are processed independently.Their applicability beyond two modalities has so far remained limited.

4 TRANSLATION

Multimodal translation maps an entity from one modality to another, such as generating a description from an image or an image from text. The survey organizes approaches into example-based and generative models, with generative methods further divided into three categories.

  • Translation: Multimodal translation generates the same entity in a different modality, including image-to-text description and text-to-image generation.The problem has early roots in speech synthesis, visual speech generation, and video description.
  • Translation: Visual scene description, including image and video captioning, became especially popular as computer vision and NLP converged and large multimodal datasets became available.Captioning serves as a test bed for multiple computer vision and NLP problems.
  • Translation: The survey distinguishes example-based models, which retrieve or combine dictionary entries, from generative models, which learn to produce translations.This distinction parallels non-parametric versus parametric machine learning approaches.
  • Example-based translation: Example-based methods can retrieve translations in unimodal or coordinated semantic spaces, with semantic-space retrieval often performing better and supporting bidirectional translation.These methods require manually constructed or learned semantic spaces and often large paired datasets.
  • Translation: Example-based translation is limited because the dictionary defines the model, making it large and inference slow, while suitable translations may be absent unless the task is simple or the dictionary is very large.Generative methods instead must understand the source and generate a target sequence or signal, complicating evaluation because many answers may be correct.
  • Generative translation: Generative models are categorized as grammar-based, encoder-decoder, and continuous generation models.Grammar-based models restrict the target domain, encoder-decoder models decode from a latent source representation, and continuous models generate the target modality.

4.3 Model evaluation and discussion

Evaluating multimodal translation is difficult because subjective tasks can have multiple valid outputs, making human judgment important but bias-prone and automatic metrics imperfect. Better evaluation is crucial for comparing approaches and defining optimization objectives.

  • Multimodal translation is difficult to evaluate because subjective tasks may have multiple correct outputs, unlike tasks with a single correct translation.Approximate automatic metrics can aid evaluation, but subjective comparisons remain challenging.
  • Human judgment is the gold standard for subjective evaluation, using Likert-scale dimensions such as naturalness, realism, correctness, relevance, order, and detail.Studies must be carefully constructed and conducted to avoid fluency, age, gender, and culture biases.
  • Automatic media-description metrics including BLEU, ROUGE, Meteor, and CIDEr measure similarity between generated and ground-truth text but have faced substantial criticism.These metrics are adapted from or based on machine-translation evaluation methods.
  • Retrieval-based evaluation ranks available captions by image fit and assesses whether correct captions receive high ranks, offering a proxy argued to better reflect human judgments.Generative captioning models can also assess caption likelihood directly.
  • Addressing evaluation is crucial for multimodal translation because it enables better comparison between approaches and better objectives to optimize.

5 ALIGNMENT

Multimodal alignment finds relationships between sub-components across modalities and is categorized as explicit alignment, whose objective is direct correspondence, or implicit alignment, which serves as a latent step for another task. The survey covers similarity-based, dynamic-programming, graphical-model, supervised, and attention-based approaches while highlighting limited annotations, difficult similarity metrics, and ambiguous correspondences as core challenges.

  • Definition: Multimodal alignment identifies relationships and correspondences between sub-components of instances from two or more modalities.Examples include linking image regions with caption phrases and aligning movies with scripts or source book chapters.
  • Alignment Types: The taxonomy distinguishes explicit alignment, which directly models cross-modal sub-component correspondences, from implicit alignment, which is learned latently for another task.Implicit alignment supports tasks including speech recognition, machine translation, media description, and visual question answering.
  • Explicit Alignment: Explicit alignment relies on similarity metrics that may be manually defined or learned, with unsupervised methods lacking direct labels and supervised methods using labeled or weakly labeled correspondences.Approaches include dynamic time warping, canonical correlation analysis, graphical models, and deep learning-based similarity measures.
  • Implicit Alignment: Implicit alignment commonly uses attention to focus decoders on targeted source sub-components, improving performance and interpretability in tasks such as question answering.Other implicit approaches learn latent alignments through similarity measures, including models linking sentence fragments with image regions for retrieval.
  • Challenges: Multimodal alignment is hindered by few explicitly annotated datasets, difficult cross-modal similarity metrics, and multiple or missing correspondences between modalities.Earlier work addressed sequence alignment mainly through unsupervised graphical models and dynamic programming with hand-defined measures.

6 FUSION

Multimodal fusion integrates information from multiple modalities to predict class or continuous outcomes, and is categorized into model-agnostic and model-based approaches. Its benefits include robustness, complementary information, and operation with missing modalities, but challenges remain in temporal alignment, supplementary information, and differing noise.

  • 6 FUSION: Multimodal fusion integrates multiple modalities to predict classification or regression outcomes.The survey distinguishes fusion as multimodal integration performed at later prediction stages, while noting that deep networks can blur the boundary with representation learning.
  • 6 FUSION: Fusion can improve robustness, capture complementary information, and preserve operation when one modality is unavailable.These benefits arise when modalities observe the same phenomenon, provide information absent from individual modalities, or allow systems to operate despite missing inputs.
  • 6 FUSION: The survey divides fusion methods into model-agnostic and model-based categories.Model-agnostic approaches are independent of a specific machine learning method; model-based approaches explicitly address fusion through kernels, graphical models, or neural networks.
  • 6.1 Model-agnostic approaches: Model-agnostic fusion comprises early, late, and hybrid approaches with distinct trade-offs in interaction modeling, flexibility, and missing-modality handling.Early fusion combines extracted features; late fusion combines unimodal decisions and supports different predictors and missing or nonparallel data; hybrid fusion seeks advantages of both.
  • 6.2 Model-based approaches: Model-based fusion includes kernel-based methods, graphical models, and neural networks designed specifically for multimodal data.Graphical models can exploit spatial and temporal structure and encode expert knowledge, while multiple kernel learning supports classification and regression but can incur slow inference and high memory use.
  • 6 FUSION: Multimodal fusion remains challenged by temporal misalignment, limited exploitation of supplementary information, and modality-specific noise that varies over time.The signals may include dense continuous streams and sparse events, while modalities can exhibit different noise types and levels at different times.

7 CO-LEARNING

Co-learning aids a resource-poor modality by exploiting knowledge from a resource-rich modality, often using the helper only during training. The section organizes approaches by parallel, non-parallel, and hybrid data and highlights co-training, representation transfer, grounding, and zero-shot learning.

  • Co-learning: Co-learning models a resource-poor modality using knowledge from a resource-rich modality, especially when annotated data, inputs, or labels are limited.The helper modality is most often used during training rather than at test time.
  • Parallel data: Parallel-data co-learning uses shared instances for co-training and representation learning across modalities.Examples include corresponding audio-video recordings and images with sentence descriptions.
  • Parallel data: Co-training bootstraps weak classifiers across modalities to create labeled samples from unlabeled data, but can produce biased samples and overfitting.The method requires parallel data because the modalities label one another’s unlabeled instances.
  • Non-parallel data: Non-parallel transfer learning improves representations by transferring information from a data-rich or clean modality to a data-scarce or noisy modality.Frome et al. coordinated CNN visual features with word2vec text representations for image classification.
  • Non-parallel data: Conceptual grounding and multimodal zero-shot learning use cross-modal representations to support semantic understanding and recognition of unseen concepts, but grounding helps only when relevant to the task.Grounding can use visual, acoustic, or olfactory signals, while multimodal ZSL recognizes objects in one modality through another modality where they were seen.
  • Hybrid data: Hybrid co-learning bridges non-parallel modalities through a shared modality or dataset, enabling coordinated representations for multilingual captioning, machine translation, and transliteration.The Bridge Correlational Neural Network uses a pivot modality when direct parallel data is unavailable.

8 CONCLUSION

The survey introduces a taxonomy of multimodal machine learning spanning representation, translation, fusion, alignment, and co-learning, and argues that it can organize future research and clarify unresolved problems.

  • 8 CONCLUSION: The taxonomy comprises representation, translation, fusion, alignment, and co-learning.Fusion has been studied for a long time, while newer interest in representation and translation has produced many multimodal algorithms and applications.
  • 8 CONCLUSION: The taxonomy is intended to catalog future research papers and clarify unresolved problems in multimodal machine learning.The authors believe it will help researchers better understand the field’s remaining challenges.
Loading 1705.09406v2…