PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

Chao Zhang, Zichao Yang, Xiaodong He, Li Deng

arXiv:1911.03977v3cs.AIcs.CLcs.CVcs.LG

TL;DR

Multimodal intelligence requires models that learn across modalities rather than handling speech, vision, or language separately. This review organizes vision–language research around representations, fusion, and applications, concluding with a synthesis of methods spanning these perspectives.

  • Problem

    Many artificial-intelligence applications involve multiple modalities, creating a need to model and learn across them.

  • Method

    The paper technically reviews multimodal representations, fusion architectures, and applications, focusing mainly on computer vision and natural language.

  • Results

    The review covers embeddings and joint spaces, operation-based fusion including attention and bilinear pooling, and applications such as captioning, text-to-image generation, and VQA.

  • Takeaways & Limitations

    The synthesis frames multimodal intelligence through shared representations, multimodal fusion, and application-specific systems for vision–language tasks.

Abstract

from arXiv · show

Deep learning methods have revolutionized speech recognition, image recognition, and natural language processing since 2010. Each of these tasks involves a single modality in their input signals. However, many applications in the artificial intelligence field involve multiple modalities. Therefore, it is of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, we provide a technical review of available models and learning methods for multimodal intelligence. The main focus of this review is the combination of vision and natural language modalities, which has become an important topic in both the computer vision and natural language processing research communities. This review provides a comprehensive analysis of recent works on multimodal deep learning from three perspectives: learning multimodal representations, fusing multimodal signals at various levels, and multimodal applications. Regarding multimodal representation learning, we review the key concepts of embedding, which unify multimodal signals into a single vector space and thereby enable cross-modality signal processing. We also review the properties of many types of embeddings that are constructed and learned for general downstream tasks. Regarding multimodal fusion, this review focuses on special architectures for the integration of representations of unimodal signals for a particular task. Regarding applications, selected areas of a broad interest in the current literature are covered, including image-to-text caption generation, text-to-image generation, and visual question answering. We believe that this review will facilitate future studies in the emerging field of multimodal intelligence for related communities.

I. INTRODUCTION

The paper frames multimodal intelligence as modeling and learning across multiple input modalities, focusing on vision–language systems. It reviews representations, fusion, and applications as three organizing perspectives.

  • Multimodal applications require models to process multiple input modalities, extending beyond advances in speech, vision, and language processing.
  • The review focuses on combining computer vision and natural language across tasks and technologies.
  • Multimodal representation learning is reviewed through supervised and unsupervised methods, including representations suited to zero-shot or few-shot learning.
  • Fusion methods are classified by their actual operations, such as attention mechanisms and bilinear pooling, because recent approaches cross conventional procedural stages.
  • Applications reviewed include image captioning, text-to-image generation, visual question answering, and visual reasoning.
  • The review also covers single-modal representations, joint multimodal spaces, missing-modality handling through zero-shot learning, and transfer from large unimodal datasets.

A. Unimodal Embeddings

Unimodal embeddings encode text, images, and speakers as learned distributed representations. These embeddings support semantic structure and downstream multimodal modeling.

  • An embedding maps a one-hot word or image-category vector to a distributed real-valued representation.
  • Distributed representations spread concept information across multiple vector elements, enabling efficient encoding in relatively low-dimensional spaces.
  • Image embeddings can be extracted from final CNN layers or designed with more direct semantic relationships.
  • Neural language models derive text embeddings by modeling word probabilities, while recurrent vectors retain information from preceding words.
  • Large-corpus language models provide general text embeddings for downstream natural-language understanding tasks with limited training data.
  • Vector arithmetic captures semantic regularities in both word and image embeddings, including gender and eyeglasses attributes in image representations.
  • Speaker representations include i-vectors and embeddings derived from neural networks trained on speaker classification.

4) Speaker representations:

The paper motivates joint embeddings because unimodal data cannot theoretically represent all human concepts. It reviews shared spaces, cross-modal alignment, and supervised factors for multimodal learning.

  • Unimodal representations may be insufficient for concepts grounded in multiple modalities, motivating joint embeddings that exploit complementary data.
  • Joint multimodal spaces can be learned by reconstructing raw inputs with multiple deep streams and shared layers.
  • Pre-trained modality-specific spaces can be mapped into a unified vector space, with fusion using addition or concatenation.
  • Multimodal training can increase similarity, correlation, or mutual information between embeddings from different modalities.
  • Fine-grained embeddings align image regions or fragments with sentence fragments, attributes, objects, relationships, and full scenes.
  • Representations can factorize into supervised multimodal discriminative factors and unsupervised intra-modality generative factors.
  • Zero-shot methods address missing or unseen categories, while correlated auto-encoders can improve one-shot and few-shot image retrieval.

4) Transformer-based methods:

Transformer-based multimodal methods extend text encoders such as BERT to incorporate visual features. The fusion review emphasizes intermediate architectures and operation-based categories.

  • 4) Transformer-based methods:: Transformers stack feedforward and multi-head self-attention blocks, avoiding the first-order Markovian assumption of RNN encoder–decoders for long sequences.
  • 4) Transformer-based methods:: BERT provides text embeddings from masked-language pretraining, motivating bimodal extensions that jointly represent text and images.
  • 4) Transformer-based methods:: Bimodal BERT variants add visual-feature tokens or introduce extra encoders and attention structures for image inputs.
  • III. FUSION: Fusion integrates unimodal representations into a compact task-specific multimodal representation.
  • III. FUSION: Intermediate fusion is reviewed because it is flexible and increasingly difficult to separate into clear procedural stages.
  • III. FUSION: The review categorizes text–image intermediate fusion as simple operation-based, attention-based, or tensor-based.
  • III. FUSION: Simple fusion operations include concatenation and weighted sums, often with few or no associated parameters.

B. Attention-based Fusion

Attention-based fusion dynamically weights modality features to produce attended representations, with variants supporting iterative, symmetric, and memory-based reasoning.

  • Attention mechanisms compute dynamically weighted sums of feature vectors to emphasize relevant information.For images, feature vectors from different regions receive different weights, producing an attended image vector.
  • Image-attention models condition visual weighting on language-processing states and use fused representations for VQA answers.An LSTM combines the current word embedding with attended image features, using its final hidden state for prediction.
  • Stacked and spatial-memory attention models query images repeatedly, refining query or memory representations across reasoning steps.Stacked attention adds previous query vectors to attended image vectors, while spatial memory aligns words with image regions over multiple hops.
  • Bottom-up and top-down attention combines salient-region proposals with question-conditioned visual attention.The bottom-up mechanism proposes salient image regions, which are then used with top-down attention.
  • Co-attention generates attended image and language vectors through parallel or alternating symmetric structures.Parallel co-attention derives both distributions simultaneously, whereas alternating co-attention cascades language-guided image attention into subsequent language attention.
  • Dual attention and related models incorporate simultaneous modality attention with iterative memory or multistage reasoning.DAN updates memory vectors across reasoning steps, while SLA retains intermediate attended information across layers.
  • High-order correlations can derive attended feature vectors for both modalities, while bimodal transformers fuse modality tokens through attention.Bimodal BERT variants distinguish word and image-fragment tokens, and OmniNet uses gated multi-head attention across modalities.

3) Attention in a bimodal transformer:

Fusion methods use gating, multiplicative interactions, and bilinear pooling to construct joint visual-textual representations, including compact and factorized approximations.

  • Gated multimodal units assign dynamically generated, dimension-specific weights to visual and textual feature vectors.The weighted sum provides a gated fusion of the two modality representations.
  • Bilinear pooling computes the outer product of visual and textual vectors, enabling multiplicative interactions between all elements.It is also called second-order pooling and differs from weighted sums, element-wise multiplication, and concatenation.
  • Compact bilinear pooling approximates polynomial-kernel interactions using count sketches and convolutions.MCB is one resulting multimodal compact bilinear method.
  • Low-rank and Tucker decompositions factorize bilinear weight tensors into lower-dimensional components and core tensors.MUTAN uses a low-dimensional core tensor and three two-dimensional weight matrices to model cross-modal interactions.
  • Bilinear pooling can be combined with attention or co-attention to form attended and joint multimodal representations.MCB and MLB can generate attended image features, while MFB and MFH support alternating co-attention.

IV. APPLICATIONS

The review covers vision-language applications including captioning, text-to-image generation, and visual question answering, with visual reasoning discussed mainly through VQA.

  • IV. APPLICATIONS: The reviewed application set includes image captioning, text-to-image generation, and visual question answering.These tasks exemplify multimodal representation learning and fusion for vision-language applications.
  • IV. APPLICATIONS: The review excludes text-based image retrieval and visual-and-language navigation because of space limitations.The scope is therefore selective rather than exhaustive across vision-language applications.
  • A. Image Captioning: Image captioning automatically generates natural-language descriptions of images and requires image understanding beyond recognition and detection.Reviewed approaches include stepwise caption generation and image-text matching.
  • IV. APPLICATIONS: Text-to-image generation produces image pixels from human-provided descriptions or keywords.The task reverses the direction of image captioning.
  • IV. APPLICATIONS: VQA maps an image and an open-ended natural-language question to an answer, requiring language understanding and reasoning capabilities.The review discusses visual reasoning methods related to VQA.
  • A. Image Captioning: End-to-end captioning commonly encodes images with CNNs and decodes captions using RNNs or LSTMs, optionally applying attention before each generated word.Attention compares the decoder state with image objects and uses weighted image features to generate the next token.
  • A. Image Captioning: Object localization, gated decoding, detected words, and external knowledge are among extensions used to enrich image captions.Localized object features improved a previous state-of-the-art model across multiple evaluation metrics.

B. Text-to-image Generation

Text-to-image generation must produce realistic images that remain semantically consistent with language. Reviewed methods progress from conditional generation toward staged refinement and layout-aware synthesis, but complex multi-object scenes remain difficult.

  • B. Text-to-image Generation: Text-to-image generation jointly requires high-quality image synthesis and language understanding.Generated images must be visually realistic and semantically consistent with their descriptions.
  • B. Text-to-image Generation: Conditional GANs use language embeddings or category information to guide image synthesis.GAN-INT-CLS produced visually plausible 64×64 images from natural-language description embeddings.
  • B. Text-to-image Generation: StackGAN decomposes synthesis into sketching and refinement by stacking two separately trained conditional GANs.The first stage generates 64×64 low-resolution images containing primitive object shapes and colors.
  • B. Text-to-image Generation: Methods improve image-text semantic consistency through alignment models, hierarchical adversarial constraints, auxiliary classifiers, and captioning-based consistency.DAMSM, HDGAN, TAC-GAN, Text-SeGAN, and MirrorGAN use different additional objectives or structures.
  • B. Text-to-image Generation: State-of-the-art methods still struggle with complex scenes containing many objects and relationships.Layout information such as keypoints, bounding boxes, and scene graphs has been introduced to control object placement and relationships.

4) Semantic layout control for complex scenes:

The review covers fine-grained and interactive control of text-to-image generation, alongside varied visual question answering formulations and datasets. It also highlights extensions toward more complex scene understanding and reasoning.

  • 4) Semantic layout control for complex scenes:: Text-to-image systems support fine-grained control through attributes, semantic layouts, natural-language editing, and dialogue-based interaction.Examples include face generation and editing, semantic modification of birds and flowers, and progressively complex scene synthesis.
  • 4) Semantic layout control for complex scenes:: Text-to-image generation has been extended from single images to multiple images and videos requiring visual consistency.
  • 4) Semantic layout control for complex scenes:: VQA asks natural-language questions about visual information, but is often simplified into classification tasks with different answer formats.Formulations include selecting among answer choices, fill-in-the-blank responses, and other classification setups based on image, question, and answer features.
  • 4) Semantic layout control for complex scenes:: VQA datasets range from DAQUAR and COCO-QA to VQA v1/v2 and Visual7W, differing in question construction, scene type, and answer format.Visual7W includes questions about objects, spatial relations, commonsense reasoning, and pointing answers linked to image bounding boxes.
  • 4) Semantic layout control for complex scenes:: Knowledge-based VQA extends standard benchmarks with questions requiring information beyond the image and question content.These extensions incorporate structured knowledge bases and require additional querying steps.

3) Integrating external knowledge sources:

The review describes external knowledge as a way to extend VQA beyond simple visual recognition toward questions requiring broader factual or commonsense information. It also discusses structured reasoning architectures for visual question answering.

  • 3) Integrating external knowledge sources:: Common VQA benchmarks often contain strong answer-distribution priors that let models overfit statistical biases instead of understanding visual scenes.GQA and VQA-CP were introduced with designs intended to reduce or challenge these biases.
  • 3) Integrating external knowledge sources:: VQA methods also face low robustness to linguistic variation, motivating datasets with rephrased questions and cycle-consistency-based approaches.
  • 3) Integrating external knowledge sources:: Additional VQA extensions address counting errors, reading text in images through OCR, low-quality user images, and rare concepts.VizWiz contains spoken-English questions from blind users, while active learning targets more informative examples.
  • 3) Integrating external knowledge sources:: Visual reasoning methods for VQA include neural module networks that parse questions into structured module networks for answer generation.Variants modify parsing, module features, or transparency of the reasoning procedure.

2) Other types of end-to-end reasoning methods:

The review covers end-to-end reasoning architectures that decompose visual questions into attended steps or model relationships among visual and textual features. It concludes by situating these methods within broader multimodal research directions.

  • 2) Other types of end-to-end reasoning methods:: MAC networks decompose questions into attended reasoning steps using recurrent cells that separate control and memory hidden states.The cells use attention and gating mechanisms to generate the hidden states.
  • 2) Other types of end-to-end reasoning methods:: Relationship networks model relationships between pairs of visual and textual features, then accumulate and transform the resulting outputs.Although they lack inductive reasoning, relationship networks achieve very high VQA accuracy on CLEVR.
  • 2) Other types of end-to-end reasoning methods:: The review organizes vision-language research around multimodal representations, multimodal fusion, and applications.It discusses embeddings, attention and bilinear-pooling architectures, captioning, text-to-image generation, VQA, and visual reasoning.
  • 2) Other types of end-to-end reasoning methods:: Future directions include constructing multimodal datasets, verifying algorithmic capabilities with commonsense-focused tasks, and updating learned commonsense knowledge.

B. Multimodal Emotional Intelligence

The review identifies multimodal emotional intelligence as a challenging direction for human-computer interaction and cognitive research. It emphasizes the need to align, reconcile, and represent emotional information across modalities within broader multimodal systems.

  • B. Multimodal Emotional Intelligence: Multimodal emotional intelligence aims to help machines understand and generate emotional content and empathize with humans.The topic is linked to anthropomorphic human-computer interaction, cognitive intelligence, and real-world applications.
  • B. Multimodal Emotional Intelligence: Key challenges include perceiving and aligning subtle emotional expressions across modalities.
  • B. Multimodal Emotional Intelligence: Multimodal emotional systems must ensure consistency and rationality of data across modalities.
  • B. Multimodal Emotional Intelligence: Another challenge is acquiring modality-invariant representations and intensities for emotions.
  • B. Multimodal Emotional Intelligence: Service-industry intelligentization requires large-scale, complex, task-oriented multimodal human-computer interaction systems.The review points to open-source frameworks, large-scale datasets, algorithm-verification platforms, and fundamental multimodal research as opportunities.
  • B. Multimodal Emotional Intelligence: Multimodal intelligence research remains in its infancy while having become an important branch of artificial intelligence development.
Loading 1911.03977v3…