PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Deep Learning

Cem Akkus, Luyang Chu, Vladana Djakovic, Steffen Jauch-Walser, Philipp Koch, Giacomo Loss, Christopher Marquardt, Marco Moldovan, Nadja Sauter, Maximilian Schneider, Rickmer Schulte, Karol Urbanczyk, Jann Goschenhofer, Christian Heumann, Rasmus Hvingelby, Daniel Schalk, Matthias Aßenmacher

arXiv:2301.04856v1cs.CLcs.LGstat.ML

TL;DR

Multimodal research must address limitations in sequence modeling and cross-modal representation learning. This book surveys multimodal architectures and reports improved photorealism and diversity in models such as GLIDE and unCLIP, while noting remaining challenges.

  • Problem

    Encoder-decoder and recurrent architectures struggle with bottlenecks, vanishing gradients, and long-distance dependencies in sequences.

  • Method

    The book synthesizes multimodal architectures, including joint text-image contrastive pretraining and automated and human evaluation of generated images.

  • Results

    GLIDE achieves more photorealistic results, while unCLIP produces greater data diversity with similar photorealism and caption similarity.

  • Takeaways & Limitations

    The reviewed text-to-image models show progress across photorealism, diversity, and caption similarity, but performance remains uneven across model capabilities.

  • Takeaways & Limitations

    Pure language models still outperform multimodal multitask models, indicating room for improvement in unified multimodal and multitask representation learning.

Abstract

from arXiv · show

This book is the result of a seminar in which we reviewed multimodal approaches and attempted to create a solid overview of the field, starting with the current state-of-the-art approaches in the two subfields of Deep Learning individually. Further, modeling frameworks are discussed where one modality is transformed into the other, as well as models in which one modality is utilized to enhance representation learning for the other. To conclude the second part, architectures with a focus on handling both modalities simultaneously are introduced. Finally, we also cover other modalities as well as general-purpose multi-modal models, which are able to handle different tasks on different modalities within one unified architecture. One interesting application (Generative Art) eventually caps off this booklet.

Technical Setup

The book was authored in Markdown, with simulations, data examples, and visualizations created in R and integrated with Markdown using rmarkdown.

  • Technical Setup: The chapters use Markdown, while R supports the simulations, data examples, and visualizations; rmarkdown combines the R code with Markdown.

0 Foreword

The foreword notes that the book was produced with the bookdown package through collaboration using Git and GitHub.

  • 0 Foreword: The book was created with bookdown, while the collaborators coordinated their work through Git and GitHub.Further details are available in the book’s repository.

1 Introduction

The book surveys state-of-the-art multimodal deep-learning methods for combining heterogeneous information and addressing unstructured, cross-modal data challenges. It progresses from NLP and computer-vision foundations to text–image architectures and general multimodal systems.

  • Motivation: Multimodal learning combines information channels such as hearing, touch, smell, taste, and sight to perceive and understand the surrounding world.The book frames multimodal deep learning as addressing challenges from unstructured data and inputs from different modalities.
  • Foundations: Chapter 2 introduces NLP and computer-vision foundations, emphasizing word embeddings and comparisons across datasets, pre-training tasks, and challenging benchmarks.NLP handles text, while computer vision handles images; word embeddings are essential in nearly all multimodal architectures.
  • Text–Image Architectures: Chapter 3 surveys architectures that combine text and images, including image captioning, image-to-text methods, image-supported language models, and text-supported vision models.Examples include Microsoft COCO, the M2 Transformer (Cornia et al., 2019), CLIP (Radford et al., 2021b), ALIGN (Jia et al., 2021a), and Florence.
  • General Multimodal Architectures: Chapter 4 extends multimodal modeling to video, speech, and tabular data, focusing on modality-independent architectures, fusion and alignment, and joint or coordinated representations.It also examines how structured modalities can be combined within general multimodal systems.

2 Introducing the modalities

NLP and computer vision have developed powerful representation-learning architectures, including sequence-to-sequence models, attention, convolutional and contrastive methods. Their convergence enables multimodal tasks such as text-to-image generation, while scarce high-quality multimodal data makes pre-training and benchmarking essential.

  • Natural Language Processing: Sequence-to-sequence models map input sequences to outputs of different lengths, supporting machine translation, video captioning, and question answering with minimal assumptions about sequence structure.Attention improves these models by focusing on relevant information, bypassing bottlenecks, and helping address vanishing gradients (Bahdanau et al. (2014)).
  • Computer Vision: Computer vision increasingly uses self-supervised contrastive learning, bringing augmented views of the same image closer while separating views from different images.SimCLR outperforms previous work, while BYOL provides another self-supervised approach for learning image representations (Chen et al., 2020a; Grill et al., 2020b).
  • Multimodal Convergence: NLP and computer vision are converging for multimodal tasks, exemplified by DALL-E 2 generating photorealistic images or art from text.This transformation from one modality to another requires multimodal datasets, which remain relatively rare.
  • Multimodal Data: Vision-language pre-training is constrained by the difficulty of obtaining high-quality image descriptions, relying on datasets including COCO, Visual Genome, Conceptual Captions, Flickr30k, LAION-400M, and LAION-5B.LAION-5B is described as the biggest openly accessible image-text dataset.
  • Benchmarking: Benchmarking compares models on specific tasks using metrics such as accuracy and can contextualize performance against human baselines.Benchmark datasets are used to test and compare models across modalities, including computer-vision datasets such as ImageNet and ImageNetReaL.

2.1 State-of-the-art in NLP

State-of-the-art NLP progresses from static word embeddings and recurrent sequence models toward attention-based Transformers and pretrained language models. These advances improve contextual representation, translation, transfer learning, and few-shot generalization while retaining limitations in long-range context, implicit knowledge, and data availability.

  • Word embeddings: Static word embeddings incorporate prior knowledge and support transfer learning, but they cannot represent a word’s context-dependent semantics.They can capture graded similarity, syntactic regularities, cross-lingual geometric mappings, and some out-of-dictionary semantic translation predictions, while distant-language translation reveals shortcomings.
  • Sequence modeling and attention: Attention considerably boosts NLP performance by bypassing encoder-decoder bottlenecks and providing shortcuts that mitigate vanishing gradients.Attention alignments also expose which source words the decoder uses when producing each target word, supporting interpretation of translation decisions.
  • Sequence modeling and attention: Recurrent models encode nearby word interactions but require O(sequence length) steps for distant dependencies, while deep LSTMs reduce perplexity by nearly 10% per additional layer.A four-layer LSTM example used 1000 cells per layer with 1000-dimensional word embeddings, representing a sentence with 8000 real numbers.
  • Transformers and pretrained language models: BERT’s masked-language pretraining and bidirectional context enabled outstanding fine-tuned results across question answering, sentiment analysis, and text summarization.Pretrained decoders are better suited than encoders such as BERT for sequence generation, while BERT remains a general-purpose tool for many tasks.
  • Transformers and pretrained language models: Transformer language models achieved successful results, with scaling exemplified by GPT-3’s 175 billion parameters compared with T5’s 11 billion.The growing size of these models makes experimentation and understanding more difficult.
  • Limitations and data efficiency: BERT representations encode hierarchical information about parts of speech, syntactic chunks, roles, and semantic preferences, but struggle with unstated visual and perceptual knowledge.Few-shot learning addresses the practical scarcity and collection cost of labelled examples across languages, domains, and tasks (Schick and Schütze, 2020).

2.2 State-of-the-art in Computer Vision

The section surveys major computer-vision advances from residual and compound-scaled convolutional networks to self-supervised contrastive learning and Vision Transformers. It emphasizes balanced scaling, stronger representation-learning techniques, and patch-based Transformers as key state-of-the-art directions.

  • Residual networks: Residual networks address degradation with identity mappings, but accuracy gains diminish in very deep models, as ResNet-1000 performs similarly to ResNet-101.Experiments found that the residual projection matrix need only match dimensions, while identity mapping is sufficient for the degradation problem.
  • EfficientNet: EfficientNet-B0 outperformed ResNet-50, while successive EfficientNet variants improved results through B7.EfficientNet compound-scales network width, depth, and resolution using fixed coefficients rather than tuning one dimension alone.
  • Self-supervised representation learning: SimCLR outperformed previous self-supervised methods using stronger augmentation, a nonlinear projection head, and contrastive cross-entropy learning without a memory bank.Stronger color augmentation particularly benefits unsupervised contrastive learning, while NT-Xent weights negatives by relative hardness through normalization, cosine similarity, and temperature.
  • Self-supervised representation learning: BYOL experiences less performance loss than SimCLR when image augmentation is limited to random crops.This comparison concerns the linear-evaluation protocol and identifies a robustness difference under weaker augmentation.
  • Vision Transformers: 88.5% accuracy: ViT-H/14 outperformed ResNet(BiT) and Noisy Student on ImageNet.ViT applies a standard Transformer to linearly embedded image patches with minimal architectural changes.

2.3 Resources and Benchmarks for NLP, CV and multimodal tasks

Improving model performance requires both stronger architectures and more data, but additional data creates labeling challenges. The broader goal is to develop generalist models that perform well across tasks without relying on huge labeled datasets.

  • Better model performance can come from improved architectures and larger or newer datasets, although newly collected data must be carefully labeled.
  • Supervised models trained on carefully labeled data perform very well on their training task but may not generalize effectively to different tasks.
  • The desired direction is generalist models that handle multiple tasks without requiring huge labeled datasets, more like humans learning efficiently with limited supervision.

2.3 Resources and Benchmarks for NLP, CV and multimodal tasks

Pre-training resources differ by modality: NLP uses text, CV images, and VL-PTM text–image pairs, while web-derived datasets introduce noise, uneven scale, and limited public availability. The section surveys benchmarks that expose weaknesses in language, vision, and vision-language evaluation, including overfitting, ambiguous labels, and remaining reasoning gaps.

  • Pre-training resources: NLP models pre-train on text, CV models on images, and VL-PTM on paired text–image data, combining characteristics of NLP and CV.
  • Pre-training resources: Text data is much more abundant than CV data, VL-PTM datasets remain relatively small but are growing quickly, and some datasets are not publicly available.Dataset access is also constrained because collected data may be proprietary or sensitive, although this does not always justify withholding it.
  • Pre-training resources: Web-derived datasets can be noisy, requiring extraction and filtering; current best practice combines large web scrapes with targeted, higher-quality datasets.Common Crawl exemplifies the scale and variability of raw web data, while the Pile was designed around diverse sources and contains about 825 GB across 22 sub-datasets.
  • NLP benchmarks: GLUE’s human leaderboard benchmark was quickly surpassed by models, motivating the creation of SuperGLUE.
  • Computer vision benchmarks: ImageNet evaluations revealed accuracy drops on newly collected, slightly harder images, while ReaL introduced reassessed labels and a metric that accommodates multiple valid image labels.Recht et al. (2019) attributed the drops to limited generalization rather than adaptivity, and Beyer et al. (2020) proposed ReaL accuracy.
  • Multimodal benchmarks: VCR remains difficult for vision-language models: humans exceed 90% accuracy, whereas released models achieved 85.5 in Q →A, 87.5 in QA →R, and 74.9 in Q →AR.At release, state-of-the-art vision models reached only 45% accuracy, indicating that visual reasoning was not solved.

3 Multimodal architectures

This section surveys multimodal architectures for images and text, covering image-to-text generation, text-to-image synthesis, visual grounding for language models, language supervision for vision, and unified models handling both modalities. It emphasizes transformer-based representation learning, self-supervision, and joint or staged integration of modalities.

  • Section overview: The chapter organizes image–text multimodal learning into five directions: Image2Text, Text2Image, Images supporting Language Models, Text supporting Image Models, and Models for both modalities.These directions span transforming one modality into the other, integrating visual information into language models, using language as supervision for vision, and building unified frameworks.
  • Image2Text: Transformer-based architectures improve captioning of complex images, with richly annotated COCO supporting meaningful Image2Text modeling (Lin et al., 2014c; Cornia et al., 2020).The section frames image description and scene-context parsing as substantially more challenging for computers than for humans.
  • Text2Image: Text2Image models encode contextual textual representations into embedding vectors that guide image generation across GAN, VAE, VQ-VAE, diffusion, and autoregressive approaches.The field is presented chronologically, with recent NLP advances enabling text-conditioned visual generation.
  • Images supporting Language Models: Visual information can support language models through either separate unimodal encoders followed by combination or simultaneous joint learning in a shared intermediate space.Silberer and Lapata (2014) use a bimodal autoencoder, whereas Bordes et al. (2020) share parameters between textual and grounded objectives.
  • Text supporting Image Models: Language supervision strengthens computer-vision training by adding signal beyond manual labels, exemplified by CLIP (Radford et al., 2021a) trained on WIT’s 400 million text-image pairs.CLIP uses large-scale web text-image data to connect language and visual representations.
  • Models for both modalities: Unified multimodal models pursue general self-supervised learning, while transformers provide scalable parallel processing and have become the de-facto standard for vision-language modeling.The section highlights transformers’ advantages over RNNs/CNNs with large datasets and discusses cross-modality and few-shot capabilities in models including data2vec and Flamingo.

3.1 Image2Text

Image captioning requires both semantic image comprehension and accurate sentence generation. The section presents MS COCO as a richly annotated resource and describes M² Transformer’s advances in image encoding, decoding, and captioning performance.

  • Image captioning: Image captioning combines semantic image comprehension with the generation of accurate and precise descriptive sentences.It is presented as a task spanning computer vision and natural language processing.
  • MS COCO: MS COCO contains 91 object categories, 328,000 images, and 2,500,000 instance labels, supporting detection, relationship modeling, and precise localization (Lin et al., 2014c).The dataset focuses on non-iconic views, semantic relationships between objects, and precise object localization.
  • MS COCO: MS COCO has fewer categories than ImageNet and SUN but the highest average number of instances per category, supporting models for precise localization (Lin et al., 2014c).Its construction emphasizes representative everyday object categories and includes many non-iconic images.
  • M² Transformer: M² Transformer learns multi-level relationships between image regions, incorporates a priori knowledge across encoder layers, and uses mesh-like decoder connectivity for language generation (Cornia et al., 2020).The architecture exploits low- and high-level visual features and applies sigmoid-gated meshed connectivity.
  • M² Transformer: M² Transformer exceeds other models on BLEU-4, METEOR, and CIDEr, while remaining competitive with SGAE on BLEU-1 and ORT on SPICE (Cornia et al., 2020).Its CIDEr score rises from 121.8 for the original Transformer to 129.2 with the reduced model and 131.2 with meshed connectivity.

3.2 Text2Image

Text-to-image models progressed from GAN-based systems with style transfer and zero-shot generation to large diffusion models with stronger photorealism, diversity, and benchmark performance. Despite these advances, compositionality, text rendering, inference speed, misuse, bias, and copyright remain important challenges.

  • 3.2 Text2Image: Earlier GAN systems transferred training-data style and background, generated unseen classes zero-shot, and generalized across MS-COCO domains, although outputs were not always coherent.GAN-INT-CLS performed best when both mentioned methods were used.
  • 3.2 Text2Image: DALL-E improved diversity and zero-shot performance over prior same-domain systems but retained low photorealism and difficulty with text and relative object positions.The comparison is presented in Figure 3.24, with qualitative outputs in Figure 3.25.
  • 3.2 Text2Image: GLIDE achieved more photorealistic zero-shot MS-COCO results than predecessors, with human preferences for photorealism and caption similarity.Its FID comparison is shown in Figure 3.26, while comparison with DALL-E 1 appears in Figure 3.27.
  • 3.2 Text2Image: UnCLIP represented greater data diversity than GLIDE at similar photorealism and caption similarity, while achieving unprecedented MS-COCO FID.The text attributes its remaining problems partly to CLIP embeddings, which improve diversity but may worsen compositional and text-rendering failures.
  • 3.2 Text2Image: Both discussed models significantly improved FID over previous work, and Parti samples were presented as the state of the art at the time of writing.The passages do not identify both models by name in the reported result.
  • 3.2 Text2Image: Text-to-image models raise risks of harassment, disinformation, embedded bias, insufficient diversity, and unresolved questions about authorship and copyright.Inpainting can intensify misuse by adding or erasing objects in real scenes, while models combine concepts from their training data.

3.3 Images supporting Language Models

This section reviews models that enhance language models with images to improve word embeddings, tracing integration from modality fusion to visual supervision and generated-image approaches. Visual grounding mainly benefits concrete concepts, while Transformer-based multimodal models generally do not surpass pure language baselines unless visual supervision is incorporated.

  • Model evolution: The models progress from concatenating textual and visual modalities toward projected visual representations, generated-image supervision, and image assignment for any token.Voken methods address low grounding coverage by pooling from a finite image set, while generated images can supervise pure language models such as BERT or visually supervised models.
  • Limitations and extensions: A one-to-one text–image assumption limits grounding because some words lack available images, motivating visual representations inferred from linguistic information or broader token-level assignment.The Voken-Classification framework can assign visual representations to any word or token in a language corpus.
  • Evaluation: Visual grounding improves representations of concrete concepts such as “cat,” “table,” and “bicycle,” but contributes little to abstract concepts.This pattern appears across intrinsic and downstream evaluations, where multimodal models can still lag behind pure language models for abstract concepts.
  • Evaluation: Transformer-based universal multimodal models generally do not outperform pure language models such as BERT or RoBERTa, whereas Voken-based visual supervision can raise performance above them.The Voken-Classification task can be integrated into pre-training, and Vokenization is identified as the main competitive form of visual supervision.
  • Few-shot learning: With up to 5% of the training set, Lu et al. (2022) showed that imagination-based supervision could outperform simple visual supervision from the Voken model, supporting few-shot multimodal learning.Few-shot evaluation addresses the high cost of collecting paired image–text data; Ive et al. (2019) also found images improved translation when textual context was generic or ambiguous.

3.4 Text supporting Vision Models

Text-supported vision models adapt NLP architectures and language supervision to computer vision, scaling image-text data to improve performance and robustness. CLIP, ALIGN, and Florence demonstrate strong results, with Florence achieving the overall top performance while web-scale data introduces noise and preprocessing needs.

  • Training approach: The models transfer NLP concepts such as Transformers and self-supervised learning to vision, jointly encoding images and text and aligning them with a contrastive objective.Natural language supervision uses image-caption co-occurrence, while contrastive alignment minimizes a symmetric loss over image-text similarities.
  • Data limitations: Web-scale image-text data removes costly manual labeling but remains extremely noisy, requiring preprocessing such as language and graphic-content filtering.The data’s noise is a central trade-off of scaling through internet-generated image-text pairs.
  • Robustness: CLIP narrows the robustness gap under distribution shift compared with an equally accurate ImageNet-trained ResNet101, whose performance deteriorates across increasingly different distributions.A ResNet101 trained on ImageNet at 76.2% accuracy retains only 32.6% on ObjectNet, motivating robustness evaluation.
  • Model comparisons: Florence has the overall top performance, beating CLIP in nearly all zero-shot and fine-tuned evaluations, while ALIGN wins half of its four reported comparisons.Florence beats CLIP by 7.3pp on original ImageNet; retrieval ranks Florence first, ALIGN second, and CLIP third.

3.5 Models for both modalities

The section presents architectures for handling text, images, and speech through unified learning frameworks, while distinguishing single-modality operation from genuine cross-modal interaction. It discusses data2vec’s self-supervised latent prediction and ViLBERT’s dual-stream co-attention for vision-language tasks.

  • Motivation and limitations: These architectures address limited multimodal data and fragmented modality-specific models through sophisticated training schemes, but data2vec still retains modality-specific encoders and uses no cross-modality training.Self-supervision and unified frameworks are motivated by the expense of labeling large multimodal datasets and the difficulty of consolidating separate models.
  • Results: Data2vec achieved state-of-the-art performance in vision and language tasks among similar self-supervised models, while ViLBERT’s dual-stream design outperformed single-stream encoding.ViLBERT performance also improved with pre-training, fine-tuning, and larger datasets.
  • Data2vec: Data2vec averages multiple teacher layers for prediction targets, improving performance across language, vision, and speech, although the optimal number of layers differs by domain.The gains are strongest in NLP; computer vision benefits up to three layers, while speech peaks at six layers.
  • Data2vec: Data2vec works with text, speech, or images through a teacher-student framework that predicts contextualized latent representations from masked inputs, reducing reliance on human labels.The teacher uses a complete input view, while the student sees a masked version; the model is not cross-modal or universal.
  • ViLBERT: ViLBERT (Lu et al., 2019b) processes vision and language in parallel streams, using co-attention to condition each modality’s attention on the other.Keys and values from each modality are fed into the other modality’s multi-head attention block.
  • ViLBERT: ViLBERT’s attention quality depends on input structure: nouns most strongly influence attention maps, and 36 image region proposals failed on one question that 72 or 108 proposals answered correctly.The evidence also suggests that removing some Wh-words can improve fine-grained final-layer attention maps.

4 Further Topics · 4.1 Including Further Modalities · 4.2 Structured + Unstructured Data

The section broadens multimodal deep learning beyond text and images toward continuous, temporally rich signals and organizes this expansion through challenges, representations, translation, architectures, and training paradigms. It also reviews structured–unstructured multimodal applications, emphasizing promising performance, interpretability, missing-data robustness, and limitations in benchmarking and data availability.

  • 4.1 Including Further Modalities: Multimodal learning must extend beyond text and static images to temporal signals such as video and audio, alongside depth, temperature, smell, touch, and balance.Text and images represent only discrete snapshots of the continuous, highly multimodal world perceived by humans.
  • 4.1 Including Further Modalities: The section organizes multimodal research around central challenges that generalize across modalities, enabling new problems to be approached without abandoning a common taxonomy.The taxonomy centers on representation spaces, alignment, fusion, translation, sampling, and retrieval.
  • 4.1 Including Further Modalities: Practical multimodal tasks can be generalized as translation: models retrieve or generate a semantically related signal in another modality, using aligned spaces for retrieval and decoders when no corresponding datapoint exists.Cross-modal retrieval is currently learned almost exclusively through contrastive learning [Chen et al. (2020b)][@ oord2018representation][@ DBLP:conf/icml/ZbontarJMLD21].
  • 4.1 Including Further Modalities: A generalized multimodal system requires both an architecture and a training paradigm, with desirable properties including input-agnostic processing, fusion, locality and compositionality preservation, and flexible outputs.Perceiver-style architectures use cross-attention to compress very large multimodal inputs, while the hierarchical variant addresses lost locality and compositionality.
  • 4.1 Including Further Modalities: Multimodal representation learning distinguishes joint representations, which merge modalities into one input and representation, from coordinated representations, which align separate modality-specific spaces through semantic similarity.Joint representations can exploit cross-modal information flow, whereas coordinated representations bring semantically related points closer across separate spaces.
  • 4.1 Including Further Modalities: The proposed unified multimodal self-supervised framework generalizes BYOL- and data2vec-inspired latent prediction to arbitrary modalities for joint and coordinated learning without contrastive methods.It uses Perceiver-compatible generic inputs and one masking strategy, removing modality-specific masking and hard negatives for cross-modal learning.
  • 4.2 Structured + Unstructured Data: Structured–unstructured multimodal models can remain interpretable while achieving high predictive performance, and end-to-end learning frequently outperforms independently learned representations or expert-information retrieval.Multimodal dropout improves robustness to missing data by dropping entire modalities during training; one study found the best performance from structured clinical plus mRNA data, while adding modalities slightly reduced performance.
  • 4.2 Structured + Unstructured Data: Evidence for structured–unstructured multimodal gains remains limited because studies often omit single-modality benchmarks, small samples constrain unstructured clinical data, and robustness concerns require cautious interpretation.Future availability of larger datasets may help deep learning methods realize their potential.

4.3 Multipurpose Models · 4.4 Generative Art

Multipurpose models combine multimodal and multitask learning to support multiple perceptions and tasks in one model, while generative-art systems create images from text prompts but raise concerns about evaluation, bias, misuse, and accessibility. Across both areas, the reviewed approaches show impressive capabilities alongside unresolved technical and societal limitations.

  • 4.3.1 Prerequisites · 4.3.1.1 Multitask Learning · 4.3.1.2 Mixture-of-Experts · 4.3.1.4 Multipurpose Models: Multipurpose models are multimodal-multitask systems that transfer knowledge across tasks and modalities to improve generalization, drawing on multitask learning and mixture-of-experts architectures.Multitask learning trains one model on multiple tasks to exploit shared patterns, while MoE architectures activate only suitable experts to increase model size efficiently.
  • 4.3.2.3 OFA - Sequentialization is All You Need: OFA outperformed SOTA models on image captioning, visual question answering, visual entailment, and visual grounding, while also transferring to described unseen tasks, though no specialist baseline was used.OFA sequentializes all inputs and produces tokenized outputs through the full transformer architecture.
  • 4.3.2.1 MultiModel · 4.3.2.2 Unified Transformer (UniT) · 4.3.2.4 Gato - A Generalist Decoder · 4.3.2.5 Comparison: None of the reviewed models besides OFA achieved SOTA overall; Gato, UniT, and MultiModel were generally comparable to specialists, while transferability appeared on low-resource or unseen tasks.Gato nevertheless underperformed competing models on ATARI Boxing, and comparisons were incomplete for some unseen or low-resource settings.
  • 4.3.1.3 Evolutionary Algorithms · 4.3.3.4 muNet (Multitask Network) · 4.3.5 Discussion: muNet uses an evolutionary algorithm to preserve previously learned knowledge during fine-tuning, while PathNet and muNet remain task-tailored and require routing methods to improve generalization.The chapter identifies scaling, routing, metrics, accessibility, bias, and environmental impact as continuing challenges for multipurpose models.
  • 4.3.3.3 LIMoE · 4.3.4 Conclusion Pathways: LIMoE combines language and vision encoders with sparse MoE layers, outperforming dense models and current SOTA models like CLIP while reducing computation through sparsity.The discussion recommends combining LIMoE-style sparsity with OFA- and Gato-style flexible encoding and developing intelligent routing for evolutionary models such as muNet and PathNet.
  • 4.4.2 How to use these models?: Text-to-image notebooks let users generate pictures from prompts without prior coding knowledge, but abstract prompts are harder to illustrate and computational limits can reduce resolution.The released smaller GLIDE model has 300 million parameters, whereas the unreleased model has about 3.5 billion parameters (Nicholas et al.).
  • 4.4.4 Discussion and prospects: Generative-art methods have improved rapidly and produce increasingly realistic images from short instructions, yet there is no direct loss function for measuring aesthetics and the artist’s role remains contested.OpenAI’s DALLE-2 (Ramesh et al., 2022a) is described as outperforming DALLE-1.
  • 4.4.4 Discussion and prospects: Realistic generated images risk plagiarism and misleading fake content, while training-data bias can shape outputs, such as associating “human being” more strongly with men than women.These concerns have contributed to some organizations withholding models or downstream models from release.

5 Conclusion

Multimodal architectures have advanced rapidly, producing increasingly realistic text-to-image outputs, but their development also raises concerns about deepfakes, computational costs, environmental burden, and unequal access. These challenges motivate safeguards and broader access to models and research.

  • Text-to-image models such as DALL-E now produce impressive and highly realistic outputs, fueling substantial public fascination and online interest.
  • Realistic generated images can be difficult to distinguish from authentic photographs, enabling deepfakes that may systematically manipulate public opinion.Joshi et al. (2021) demand automated tools capable of detecting such manipulated media.
  • Larger architectures can improve performance, but training and storing their parameters require substantial time, computational power, storage, and financial resources.Training GPT-2 took about 168 hours on 32 TPUv3 chips and was estimated to cost $12,902–$43,008 (Strubell et al., 2019c).
  • The hardware, technology, and funding required to run advanced models are often concentrated among large technology companies, creating unequal access between academia and industry.Companies may also withhold their best models because they are products and contribute to intellectual property.

6 Epilogue

The booklet offers a non-exhaustive overview of Multimodal Deep Learning because of its limited scope and the field’s rapid development. It closes by contextualizing emerging work, including Stable Diffusion and text-to-video generation.

  • Scope and limitations: The booklet cannot exhaustively cover Multimodal Deep Learning because the project was completed within a limited timeframe and represented about one third of a semester’s ECTS workload.
  • Scope and limitations: Because the field is advancing rapidly, some architectures, improvements, and ideas had not been published when the chapter topics were selected, although ongoing research was incorporated in some cases.
  • Emerging generative models: Stable Diffusion, introduced by researchers from Björn Ommer’s group at LMU, generates photorealistic images from free-text prompts and extends the generative models discussed earlier.
  • Emerging generative models: Research is extending natural-language image generation toward video, including an Imagen extension developed by Google Research’s Brain Team.
Loading 2301.04856v1…