Multimodal Deep Learning

Technical Setup

• rmarkdown was used to combine R code and Markdown.Source: Technical Setup · p. 7
• Simulations, data examples, and visualizations were created with R.Source: Technical Setup · p. 7
• The book chapters were written in Markdown.Source: Technical Setup · p. 7

0 Foreword

• Contributors collaborated using git and github.Source: §0 · p. 8
• The book used the bookdown package.Source: §0 · p. 8
• Readers are directed to the book’s repository for further details.Source: §0 · p. 8

1.1 Introduction to Multimodal Deep Learning

• Multimodal deep learning must represent unstructured inputs numerically and determine how to combine information across modalities.Embeddings encode information as vectors in a latent space; text-to-image generation additionally requires relating contextual word relationships to spatial pixel relationships.Source: §1.1 · p. 9

1.2 Outline of the Booklet

• The booklet surveys state-of-the-art multimodal methods, establishing language and vision foundations before examining architectures that combine them.Chapter 2 introduces word embeddings, transformer attention, and vision architectures, alongside datasets, pre-training tasks, and benchmarks.Source: §1.1 · p. 10, §1.2 · p. 10
• Chapter 3 examines image captioning and text-to-image generation, covering adversarial networks, variational autoencoders, transformers, and text-guided diffusion models.These tasks illustrate transforming image information into text and generating images from short text prompts.Source: §1.2 · p. 10
• Further architectures use images to support language, text to support vision, or process both modalities together.Mechanisms include sequential and grounded embeddings, contrastive text–image learning, zero-shot classification, and co-attention. Other approaches share learning across speech, vision, and language or use few-shot learning with frozen pretrained vision and language models.Source: §1.2 · pp. 10–11
• Chapter 4 extends the discussion to other modalities, structured–unstructured data fusion, multipurpose models, and generative art.It frames general architectures around fusion, alignment, and joint versus coordinated representations, including video, speech, and tabular data. Structured–unstructured fusion is illustrated through survival analysis and economics.Source: §1.2 · p. 11

Natural language processing

• Word embeddings (Mikolov et al. (2013a), Bojanowski et al. (2016)) encode semantics; encoder–decoders (Sutskever et al. (2014)) map different-length sequences.Embeddings place semantically similar words near one another in a lower-dimensional space. Encoder–decoders make minimal assumptions about sequence structure, accommodating different word orders and supporting translation, video captioning, and question answering.Source: §2 · p. 13
• Attention, introduced by Bahdanau et al. (2014), lets models focus on relevant information and improves tasks such as machine translation.Direct decoder access to the source avoids the bottleneck and provides shortcuts to distant states, helping address vanishing gradients.Source: §2 · p. 13
• Transformers (Vaswani et al. (2017b)) use attention without sequential input processing, supporting parallel computation and retention of context in long sequences.Pre-trained architectures such as BERT, T5, and GPT-3 (Brown et al., 2020) can be fine-tuned for specific language tasks.Source: §2 · p. 13

Computer vision

• Image classification research addresses residual learning and network scaling, while standard Transformers are also applied directly to images.Residual learning addresses training-accuracy degradation, while EfficientNets use compound scaling across width, depth, and image resolution. Specialized attention patterns have hindered effective hardware scaling of some convolution-replacing architectures.Source: §2 · p. 14
• Self-supervised vision methods include SimCLR (Chen et al., 2020a), SwAV (Caron et al., 2020), and BYOL.SimCLR outperforms previous work; SwAV learns by predicting one view’s assignment from another view’s representation. Contrastive methods bring augmented views of the same image closer while separating views from different images.Source: §2 · p. 14

Multimodal data and evaluation

• Multimodal modeling depends on scarce paired data, large-scale pre-training, task-specific fine-tuning, and benchmarks for comparing models.DALL-E 2 illustrates text-to-image generation; vision-language pre-training datasets include COCO, Visual Genome, Conceptual Captions, Flickr30k, LAION-400M, and LAION-5B. Benchmarks evaluate language, vision, and cross-modal tasks using task-specific metrics and can provide human baselines.Source: §2 · p. 15

2.1.2 Word Embeddings

• Word embeddings replace sparse, high-dimensional word representations with trainable dense vectors that place semantically similar words near one another.Classical representations grow with vocabulary size and fail to capture similarity, limiting generalization and handling of rare words. Embedding arithmetic captures syntactic and semantic relationships, including the king–man–woman–queen analogy.Source: §2.1.2 · pp. 17–18
• Static embeddings incorporate prior knowledge but cannot distinguish a word’s context-dependent meanings.Traditional embeddings assign precomputed representations, so different senses of “bank” share a representation. Contextual embedding methods address this limitation.Source: §2.1.2 · pp. 17–18
• Almost 90 % precision was achieved by linearly mapping English and Spanish embedding spaces for translation.Geometric distance can flag dictionary errors, while training embeddings on large corpora supports sensible out-of-dictionary predictions. English–Vietnamese translation was less accurate because word correspondence and numerous synonyms complicated exact predictions (Mikolov et al. (2013c)).Source: §2.1.2 · p. 19
• Word2vec’s CBOW predicts words from context, while skip-gram predicts surrounding words from the current word (Mikolov et al. (2013a)).Both models use softmax output classifiers. Bojanowski et al. (2016) incorporate morphology into skip-gram, while GloVe combines local context with global co-occurrence statistics without a neural network.Source: §2.1.2 · p. 20

2.1.3 Encoder-Decoder

• Nearly 10% lower perplexity per additional layer distinguished deep from shallow LSTMs in sequence-to-sequence models.Encoder-decoder architectures encode variable-length inputs into a fixed context and decode variable-length outputs. In Cho et al. (2014), the context summarizes the input and conditions each decoder hidden state.Source: §2.1.3 · pp. 21, 23

2.1.4 Attention

• Attention bypasses the fixed-vector bottleneck and mitigates vanishing gradients by accessing relevant encoder states, improving NLP performance.Bahdanau et al. (2014) jointly learn alignment and translation rather than relying on one vector to summarize arbitrarily long inputs. Softmax-normalized scores weight encoder states, and the attention output is concatenated with the decoder state.Source: §2.1.4 · pp. 23–24
• Attention alignments reveal which source words influence each generated word and expose translation reorderings (Bahdanau et al. (2014)).English–French alignments are largely monotonic, with exceptions reflecting different adjective–noun ordering. Qualitative analysis found that model alignments predominantly matched intuition.Source: §2.1.4 · pp. 25–26

2.1.5 Transformer

• Transformers replace recurrence with attention, enabling parallelizable pretraining and strong performance across language tasks.Recurrent models require O(sequence length) steps for distant words to interact and restrict parallel computation. Self-attention connects states within the same sequence, returning values weighted by query–key matches.Source: §2.1.5 · pp. 26–28
• Vaswani et al. (2017c) combine parallel self-attention with feed-forward nonlinearities, positional representations and masking to address attention’s limitations.Interactions between all words at every layer give O(1) maximum interaction distance and bidirectional context. Positional representations restore word-order information, while masking prevents access to future words when the objective requires it.Source: §2.1.5 · p. 29
• Multi-head attention, residual connections, layer normalization and scaled dot products support deep transformer encoder–decoder stacks.Independent attention heads combine feature-specific outputs, while scaled attention divides query–key dot products by d_k^0.5 before softmax to control extreme scores. Decoder masking sets future scores to −∞ for parallel processing, and a final linear layer followed by softmax produces vocabulary probabilities.Source: §2.1.5 · pp. 30–31

2.1.6.1 BERT

• 15% of sequence tokens are prediction targets in BERT’s masked language modelling, preserving bidirectional context.Of those targets, 80% become MASK, 10% become random tokens and 10% remain unchanged; next-sentence prediction was dropped in later work for insignificant impact. Fine-tuned BERT supports question answering, sentiment analysis and summarization, but pretrained decoders outperform pretrained encoders for sequence generation.Source: §2.1.6.1 · pp. 33–34

2.1.6.2 T5

• T5 unifies NLP tasks as text-to-text prediction, using C4 pretraining and span corruption.Variable-length spans are replaced with unique placeholders for reconstruction, while the decoder retains a language-modelling objective. The same model, loss and hyperparameters support translation, summarization, question answering, classification and even regression through number strings.Source: §2.1.6.2 · p. 34

2.1.6.3 GPT-3

• 175 billion parameters and around 500 billion training tokens characterize GPT-3, whose in-context examples enable task learning without gradient steps.GPT uses stacked decoders (Brown et al., 2020), pretrained to predict the next word conditioned on preceding words. The largest T5 model has 11 billion parameters, illustrating the scale comparison presented here.Source: §2.1.6 · p. 33, §2.1.6.3 · pp. 34–35

2.1.7.1 Concerns regarding growing size of Language Models

• 249t of CO2 was estimated for training a Transformer (big) model by Strubell et al. (2019a).An estimated 0.1 BLEU improvement in English-to-German translation increased computation costs by $150,000. Growing models require larger, often poorly documented datasets; web-data biases and disproportionate environmental harms disadvantage marginalized communities.Source: §2.1.7.1 · pp. 35–36

2.1.7.2 Improving Understanding of Transformer-based models

• BERT encodes hierarchical syntax (Lin et al., 2019, Liu et al. (2019a)) but struggles with perceptual knowledge.It exhibits semantic knowledge but fails some inferences. Although model size hinders experimentation, attention weights aid interpretation (Clark et al., 2019); most heads can be removed without notable performance loss, and attention maps reveal frequent focus on SEP.Source: §2.1.7.2 · pp. 36–37

2.1.7.3 Few-Shot Learning

• Few-shot learning addresses scarce, costly labels with limited demonstrations and lower computational and environmental costs.Brown et al. (2020) describe forward-only prompting with task descriptions, examples and prompts, with few-shot performance improving as parameter counts grow. Example wording, order, decoding and hyperparameters can inflate apparent ability; Lialin et al. (2022) report compositional zero-shot failures without a parameter-count correlation, suggesting optimization or masking strategies may matter more.Source: §2.1.7.3 · pp. 37–38

2.2.1 History

• 16.4% error marked AlexNet’s breakthrough at the 2012 ImageNet recognition challenge.Earlier neurophysiological research described hierarchical visual processing, inspiring computer scientists to recreate neural structures.Source: §2.2.1 · pp. 39–40

2.2.2 Supervised and unsupervised learning

• Supervised learning (Education, 2020a) uses labels for classification or regression, whereas unsupervised learning (Education, 2020b) discovers patterns in unlabeled data.Self-supervision learns one input part from another (techslang, 2020), and contrastive learning identifies similar and dissimilar unlabeled examples (Tiu, 2021). Semi-supervised learning combines labeled and unlabeled data, helping extract features from complex, high-volume datasets.Source: §2.2.2 · pp. 40–41

2.2.3 Scaling networks

• ConvNets can scale depth, width, or image resolution; greater width and resolution help capture finer-grained features.Depth scaling motivated ResNet, while EfficientNet scales all three dimensions.Source: §2.2.3 · p. 41

2.2.4.1 Deep Residual Learning

• Residual learning addresses depth-related degradation by learning F(x) = H(x) − x and reconstructing the mapping through shortcut addition F(x) + x.Degradation increases training error as depth grows and is not caused by overfitting. Identity shortcuts suffice to address it without extra parameters or substantial computation; projections match differing dimensions.Source: §2.2.4.1 · pp. 42–43, §2.2.4 · p. 42

2.2.4.2 Network Architectures

• Residual architectures add shortcuts to VGG-inspired baselines (Simonyan and Zisserman, 2014), using zero-padding or 1×1 convolutions to match dimensions.The described plain baseline has 34 weighted layers, stride-two downsampling, global average pooling, and a 1000-way fully connected softmax output.Source: §2.2.4.2 · pp. 43–44, §2.2.4.1 · p. 43

2.2.5 EfficientNet

• EfficientNet balances depth, width, and resolution through fixed scaling coefficients rather than enlarging only one dimension.The reported comparison finds better accuracy at the same FLOPS when width scaling accompanies increased depth and resolution, supporting coordinated scaling.Source: §2.2.5 · p. 44, §2.2.5.1 · p. 48

2.2.5.1 Compound Model Scaling

• Compound scaling preserves baseline layer operators while coordinating depth, width, and resolution through a resource-controlled coefficient ϕ.Single-dimension gains diminish: ResNet-1000 has similar accuracy to ResNet-101, while wide, shallow networks struggle with higher-level features. Grid-searched constants allocate resources; FLOPS scale proportionally to d*w^2*r^2, with α*β^2*γ^2 ≈ 2 yielding approximately 2^ϕ growth.Source: §2.2.5.1 · pp. 46–48

2.2.5.2 EfficientNet Architecture

• EfficientNet-B0 is a mobile-sized baseline developed to demonstrate compound scaling, with efficacy assessed using accuracy and FLOPS.Baseline design matters because scaling does not change the network’s layer operators.Source: §2.2.5.2 · p. 48
• EfficientNet searches scaling coefficients once on a small baseline, then holds them fixed while varying ϕ to construct EfficientNet-B1 through B7.With ϕ = 1, the search selects α = 1.2, β = 1.1, and γ = 1.15 under α · β^2 · γ^2 ≈ 2. Searching around larger models could improve performance further but becomes prohibitively expensive.Source: §2.2.5.2 · pp. 48–49

2.2.5.3 Results and comparison of the networks

• EfficientNet-B0 outperforms ResNet-50, and EfficientNet-B1 outperforms ResNet-152 on ImageNet, supporting scaling across three dimensions rather than depth alone.Successive EfficientNet variants improve performance, but computational demands are a drawback. ResNet-34 improves top-1 accuracy by 3.5 % over plain-34, and increasing ResNet depth further improves accuracy.Source: §2.2.5.3 · pp. 49–50

2.2.6 Contrastive learning

• Bootstrap Your Own Latent bootstraps image representations to avoid doubled datasets and unnecessary image comparisons.This addresses the computational burden described for SimCLR (Chen et al., 2020a), within efforts to reduce expensive, time-consuming human labeling.Source: §2.2.6 · pp. 50–51

2.2.6.1 A Simple Framework for Contrastive Learning of Visual Representations

• SimCLR (Chen et al., 2020a) learns visual representations without human supervision or a memory bank and reports outperforming previous work.Effective representations benefit from combined augmentations, nonlinear projection, normalized embeddings, and suitable temperature scaling. Larger batches, longer training, and deeper or wider networks also improve contrastive representation learning.Source: §2.2.6.1 · p. 51

2.2.6.2 The Contrastive Learning Framework

• SimCLR (Chen et al., 2020a) combines augmented-view pairs, a ResNet-50 encoder, a nonlinear projection head, and NT-Xent contrastive loss.Stronger color augmentation improves unsupervised linear evaluation but not supervised performance, and nonlinear projection outperforms linear or absent projection. Normalization and temperature scaling weight hard negatives; omitting them worsens learned representations despite higher contrastive-task accuracy.Source: §2.2.6.1 · p. 51, §2.2.6.2 · pp. 52–54

2.2.6.3 Bootstrap Your Own Latent; 2.2.6.4 Comparison of contrastive learning frameworks

• 74.3 top-1 accuracy and 91.6 top-5 accuracy for BYOL’s ResNet-50 exceed SimCLR’s ResNet-50 scores of 69.3 top-1 accuracy and 89 top-5 accuracy.BYOL matches augmented views without negatives using an online-only predictor and an exponential-moving-average target network. Wider encoders improve both frameworks; BYOL also loses less performance than SimCLR when augmentation uses only random crops.Source: §2.2.6.3 · pp. 54–55, §2.2.6.4 · p. 56

2.2.7 Transformers in Computer Vision

• Vision Transformers treat linearly embedded image patches as tokens, applying a minimally modified standard Transformer to supervised image classification.Earlier convolution–attention hybrids and convolution replacements faced difficulties scaling specialized attention patterns effectively on modern accelerators.Source: §2.2.7 · p. 57

2.2.7.1 Vision Transformers

• Vision Transformers use learnable class and positional embeddings with a standard self-attention encoder, transferring large-dataset pretraining to smaller downstream tasks.Images become flattened patches, usually 16 × 16; fine-tuning replaces the classification head and can benefit from higher resolution. Compared with CNNs, image-specific inductive bias is limited, chiefly entering through patch extraction and resolution adjustment.Source: §2.2.7.1 · pp. 57–59
• 88.5% ImageNet accuracy for ViT Hughe with 14×14 patches exceeded ResNet BiT’s 87.54% and Noisy Student’s 88.4%.ViT Large with 16×16 patches achieved 87.76% accuracy. The ResNet(BiT) comparator replaced batch normalization with group normalization and used standardized convolutions.Source: §2.2.7.1 · p. 59
• ViT outperformed CNN-based architectures on all larger datasets but performed slightly worse than CNN networks on a smaller dataset.ImageNet was used to explore model scalability.Source: §2.2.7.1 · p. 59

2.2.8 Conclusion

• The conclusion describes the reviewed networks as highly accurate and difficult to outperform, with practical applications directing computer-vision network and framework development.These applications support humans in everyday tasks.Source: §2.2.8 · p. 59

• Expensive human labeling makes supervised learning a bottleneck for further progress.Better architectures and additional data can improve models, but new data require careful human labeling, whose cost increases with data volume.Source: §2.3 · p. 60
• Supervised models can perform well on their trained task yet poorly on others, while labeling everything in the world is infeasible.Source: §2.3 · p. 60
• Meta AI identifies self-supervised learning as a promising route toward background knowledge and common sense in AI.Self-supervised learning uses a supervised learning algorithm without an external supervisor; the broader goal is generalist models that perform well across tasks without huge labeled datasets.Source: §2.3 · p. 60

2.3.1 Datasets

• Pre-training uses text for NLP, images for CV, and image–text pairs for vision–language models, whose descriptions can be richer than classification labels.Internet crawling can yield noisy data, including uninformative image alt-text. Text corpora are larger than CV and vision–language resources, while private datasets hinder reproducibility.Source: §2.3.1 · p. 61

2.3.1.1 Natural Language Processing Datasets

• Rosset (2020) recommends combining large web scrapes with targeted, higher-quality datasets, an approach addressed by the Pile.The Pile combines 22 sub-datasets totaling around 825 GB; mC4 supplies multilingual resources spanning more than 100 languages. Common Crawl requires careful extraction and filtering, while Bandy and Vincent (2021) identify BooksCorpus concerns involving copyright, duplicates, skewed representation, and problematic content.Source: §2.3.1.1 · pp. 62–64

2.3.1.2 Computer Vision Dataset

• ImageNet dominates CV pre-training; larger proprietary JFT and EFT datasets face label-quality, transparency, and computational limitations.JFT-300M contains 300 million images, approximately 20% noisy labels, and a heavily long-tailed class distribution. EFT's 100k classes entail large models and slow training, while neither proprietary dataset has a published datasheet.Source: §2.3.1.2 · pp. 64–65
• More than 10 million manually labeled bounding boxes make Objects365 useful for localization, while COCO supports vision and multimodal tasks.Objects365 covers 365 categories and over 600K training images. COCO targets non-iconic scenes (Lin et al., 2014b), adding instance segmentation and five captions per image.Source: §2.3.1.2 · pp. 65–66

2.3.1.3 Multi Modal Datasets

• 5.85 billion CLIP-filtered image–text pairs comprise LAION-5B, extending the open resources established by LAION-400M.LAION-400M contains 400 million pairs and removes samples with image–text similarity below 0.3 using CLIP embeddings, with its construction procedure openly described. Public datasets with clear, permissive licenses can serve academia and industry, but access remains uneven.Source: §2.3.1.3 · pp. 67, 69
• 650 million image–text pairs comprise WuDaoMM, while Localized Narratives synchronizes images, text, speech, and grounding.WuDaoMM releases a Chinese base dataset containing about 5 million high-quality, strongly correlated examples across 19 categories. Localized Narratives aligns words with image regions through simultaneous spoken descriptions and mouse movements, with manual transcription compensating for imperfect speech recognition.Source: §2.3.1.3 · pp. 67–68
• More than 37.6 million image–text sets across 108 languages make WIT (Srinivasan et al., 2021) a resource addressing English-centric multimodal dataset coverage.These span 11.5 million unique images, with more than 12K examples per language and more than 100K image–text pairs in 53 languages. Human annotation judged 98.5% of randomly sampled image–text associations favorably.Source: §2.3.1.3 · pp. 68–69

2.3.1.4 Bias In Datasets

• Internet-derived datasets overrepresent younger users and developed countries, and GPT-2 examples illustrate how gender bias appears in generated text.Sheng et al. (2019) show continuations associating a man with car sales and a woman with prostitution.Source: §2.3.1.4 · pp. 69–70
• Dataset biases can reappear in model decisions, while retracted datasets and problematic multimodal training data remain concerns.TinyImages and MS-Celeb remain available through file sharing despite retractions. LAION-400M's CLIP-based curation potentially propagates CLIP biases (Radford et al., 2021a); problematic content identified by Birhane et al. (2021), and LAION-5B's shared pipeline, motivate internal and external audits.Source: §2.3.1.4 · pp. 70–71

2.3.2 Pre-Training Tasks

• Contrastive learning requires hard negatives; BYOL instead learns without negative pairs using a moving-average target encoder.Energy-based models score input incompatibility, and contrastive learning separates incompatible pairs to prevent collapse; CLIP (Radford et al., 2021a) and ALIGN exploit image-text pairs, with CLIP providing readily available negatives. BYOL predicts a lagged version of its outputs but still depends on human-designed augmentations.Source: §2.3.2 · pp. 72–73
• 75% of image patches are masked by MAE, versus 15% of data masked by BERT.An encoder processes visible patches and a decoder reconstructs masked pixels, producing rich representations despite blurry reconstructions. Subsequent supervised fine-tuning updates pretrained weights for downstream tasks but requires large labeled datasets for each task.Source: §2.3.2 · p. 73

2.3.3 Benchmarks

• 2.2% median and 3.2% average overlap appeared in CLIP's analysis; overall accuracy rarely shifted by more than 0.1% (Radford et al., 2021a).Held-out benchmarks can share training biases and overestimate real-world performance, while differing pre-training datasets complicate model comparisons. Zero-shot benchmarks test generalization to classes unseen during training.Source: §2.3.3 · pp. 73–74

2.3.3.1 Natural Language Processing Benchmarks

• Nine GLUE task datasets cover single-sentence, similarity/paraphrase, and inference tasks; models surpassed its human benchmark, motivating the more challenging eight-task SuperGLUE.SuperGLUE adds diverse task formats, comprehensive human baselines, a single-number metric, and an analysis toolkit. Both suites largely reduce evaluation to classification, raising questions about their coverage of general language understanding.Source: §2.3.3.1 · pp. 74–76
• Over 50,000 adversarially written unanswerable questions distinguish SQuAD 2.0 from SQuAD's guaranteed-answer setting.SQuAD contains 107,785 question-answer pairs across 536 Wikipedia articles and requires selecting answer spans rather than choosing supplied answers. Non-English versions and multilingual resources such as XQuAD, MLQA, and TyDi extend question-answering evaluation across languages.Source: §2.3.3.1 · p. 77
• More than 200 tasks in BIG-bench probe capabilities beyond narrowly targeted benchmarks; GPT-3 and PaLM perform poorly relative to expert humans.This living benchmark accepts reviewed community submissions, including small tasks that broaden topical and multilingual coverage. Tasks primarily evaluate pretrained models without task-specific fine-tuning in zero- and few-shot settings, addressing narrow coverage and short benchmark lifespans.Source: §2.3.3.1 · pp. 77–78
• Reliable NLP evaluation should test robustness, annotation quality, statistical power, and harmful biases rather than reward unreliable systems (Bowman and Dahl, 2021).Machine translation is evaluated through WMT shared tasks, with submissions ranked by human evaluation and BLEU measuring similarity to human reference translations. Tasks include bilingual translation and using a third language to improve translation between two others.Source: §2.3.3.1 · pp. 78–79
• CheckList evaluates NLP behaviors beyond held-out accuracy through minimum-functionality, invariance, and directional-expectation tests.Ribeiro et al. (2020) introduced this open-source, model-agnostic and task-agnostic methodology, combining linguistic capabilities with test types to generate diverse cases. Minimum-functionality tests check simple behaviors, invariance tests require unchanged predictions under label-preserving perturbations, and directional tests require specified prediction changes.Source: §2.3.3.1 · p. 79

2.3.3.2 Computer Vision Benchmarks

• ImageNet-V2 exposed generalization failures, while ReaL labels address single-label evaluation that can penalize valid descriptions of images containing multiple objects.Recht et al. (2019) found accuracy drops across all tested models, attributing them to slightly harder images rather than adaptivity. Beyer et al. (2020) introduced ReaL accuracy, which accepts a top-1 prediction whenever it belongs to the image’s reassessed label set.Source: §2.3.3.2 · pp. 79–80
• COCO and Object365 benchmark object detection, while COCO and ADE20K support segmentation evaluated through task-specific overlap metrics.Detection uses mean Average Precision at IoU = .50:.05:.95; instance segmentation separates individual objects, semantic segmentation labels pixels by class, and panoptic segmentation combines both. ADE20K uses mean IoU across classes, with segmentation IoU calculated as TP/(TP + FN + FP).Source: §2.3.3.2 · pp. 80–82

2.3.3.3 Multi-Modal Benchmarks

• 85.5 in Q →A, 87.5 in QA →R, and 74.9 in Q →AR were the best VCR scores reported at writing.VCR evaluates answer selection and rationale selection, requiring both to be correct for Q →AR; at release, humans exceeded 90% accuracy while state-of-the-art vision models achieved 45%. To reduce language-prior shortcuts, VQA v2.0 associates questions with complementary images requiring different answers (Goyal et al. (2017)).Source: §2.3.3.3 · pp. 83–84

2.3.3.4 GQA

• GQA offers more objective, unambiguous, compositional, image-answerable questions than VQA 2.0, with more thorough coverage per image.Hudson and Manning (2019) introduced GQA with 113K images and 22M questions spanning recognition, relations, spatial reasoning, logical inference, and comparisons. Its additional metrics assess Consistency across responses, Validity within the question’s scope, and Plausibility of the answer.Source: §2.3.3.4 · p. 84
• Generative benchmarks assess image quality and prompt alignment, but cherry-picked outputs may not represent single-shot performance.Lower FID indicates closer generated–real image distributions; caption-based alignment uses BLEU, CIDEr, METEOR, and SPICE, alongside human judgments of quality and prompt fit. PartiPrompts supplies over 1600 English prompts across categories and difficulty levels, while DrawBench and long Localized Narratives prompts provide additional evaluations.Source: §2.3.3.4 · pp. 85–86
• Capability-level understanding remains underexplored; FOIL-COCO and VALSE diagnose image–language mismatches and linguistic grounding.FOIL-COCO, introduced by Shekhar et al. (2017), inserts one caption error and tests caption classification, erroneous-word detection, and correction. VALSE provides task-independent, zero-shot tests of existence, plurality, counting, spatial relations, actions, and coreference using language-model-generated, inference-filtered, human-validated foils.Source: §2.3.3.4 · pp. 86–87

2.3.3.5 Other Benchmarks

• 284t of CO2 was estimated for training a large Transformer with neural architecture search by Strubell et al. (2019b).Training BERT base on GPUs without hyperparameter tuning was estimated to require energy comparable to a trans-American flight. Online energy-accounting tools and initiatives such as SustainNLP encourage tracking environmental costs and prioritizing computationally efficient hardware and algorithms.Source: §2.3.3.5 · p. 87

Image2Text

• The Meshed-Memory Transformer improves image encoding and language generation for image captioning on MS COCO (Cornia et al., 2020).MS COCO (Lin et al., 2014c) addresses scene understanding through non-iconic object detection, segmentation, and captioning, and supports comparisons among fully attentive captioning models.Source: §3 · p. 89

Text2Image

• Text2Image uses contextual text embeddings from transformer-like encoders to guide image generation.The chronological overview covers generative adversarial networks, variational autoencoders, vector-quantized variational autoencoders, diffusion, and autoregressive models as foundations of recent text-to-image advances.Source: §3 · pp. 89–90

Images supporting Language Models

• Visual grounding supports language models at word/token and sentence levels, often improving performance over text-only models while remaining challenged by abstract words and sentences.Silberer and Lapata (2014) combine separately encoded textual and visual representations through a bimodal autoencoder, assuming one-to-one correspondence between spaces. Bordes et al. (2020) instead share parameters between textual and grounded objectives, projecting sentence embeddings into an intermediate grounded space without requiring that correspondence.Source: §3 · p. 90, Figure 3.1 · p. 91

Text supporting Image Models

• CLIP (Radford et al., 2021a) enables zero-shot visual classification competitive with task-specific supervised models without seeing the target dataset.Using WIT’s 400 million internet-scraped text-image pairs, CLIP jointly pretrains image and text encoders to identify matching pairs within batches, more efficiently than generating captions. At test time, the learned text encoder embeds target classes to support zero-shot classification.Source: §3 · p. 91

Models for both modalities

• data2vec uses one self-supervised framework to predict full-input latent representations from masked speech, language, or visual inputs through self-distillation (Baevski et al. (2022)).Unified frameworks aim to reduce modality-specific biases and improve robustness; unlike data2vec, VilBert (Lu et al. (2019b)) supports cross-modality tasks. Flamingo (Alayrac et al. (2022)) has 80B parameters and uses a large language model for few-shot, open-ended tasks, demonstrating perceiver architectures (Jaegle et al. (2021a)) and communication between pretrained, frozen models.Source: §3 · p. 92

3.1.1 Microsoft COCO: Common Objects in Context

• MS COCO contains 91 common object categories, 328,000 images, and 2,500,000 instance labels, addressing non-iconic views, semantic relationships between objects, and precise object localization (Lin et al., 2014c).Categories without clear boundaries, such as sky, sea, and grass, are excluded; complex images containing multiple objects are gathered to support detection of contextual relationships.Source: §3.1.1 · p. 93

3.1.1.1 Image Collection and Annotation for MS COCO

• MS COCO combines Flickr keyword collection and paired-category searches with category labeling, instance spotting, and instance segmentation (Lin et al., 2014c).Category labeling groups 91 categories into 11 super-categories and uses eight workers per image. Segmentation workers undergo category-specific training, and every segmented instance receives explicit verification.Source: §3.1.1.1 · pp. 95–96

3.1.1.2 Comparison with other data sets

• 3.5 categories and 7.7 instances per image characterize MS COCO, exceeding ImageNet and PASCAL VOC but remaining below SUN (Lin et al., 2014c).MS COCO has the highest average instances per category among these datasets, despite fewer categories than ImageNet and SUN. Compared with PASCAL VOC, it has more categories and more instances per category.Source: §3.1.1.2 · pp. 96–97, Figure 3.4 · p. 97

3.1.1.3 Discussion

• Over 70,000 worker hours supported annotation of 2.5 million everyday-object instances for MS COCO (Lin et al., 2014c).The discussion presents MS COCO as a benchmark for object detection, segmentation, and scene understanding. Adding “stuff” labels, such as sky and grass, is proposed to provide additional contextual information.Source: §3.1.1.3 · pp. 97–98

3.1.2 Models for Image captioning

• CNN–RNN captioning pipelines precede fully attentive M 2 captioning,, (Cornia et al., 2020).Earlier pipelines use CNNs to encode visual information and RNNs to generate language. The review identifies limitations in RNN representation power and sequential processing (Cornia et al., 2020).Source: §3.1.2 · p. 98, §3.1.3 · p. 99

3.1.3 Meshed-Memory Transformer for Image Captioning

• The Meshed-Memory Transformer combines persistent memory for prior knowledge with gated connections between multiple encoder and decoder layers (Cornia et al., 2020).Its encoder represents both low- and high-level relationships between image regions, which the decoder combines during sentence generation.Source: §3.1.3 · pp. 99–100

3.1.3.1 M 2 Transformer Architecture

• M 2 augments attention keys and values with image-independent learned memory, while its decoder accesses every encoder layer through gated cross-attention (Cornia et al., 2020).Both modules contain multiple layers, and the decoder generates captions word by word. Masked self-attention restricts predictions to previously generated words.Source: §3.1.3.1 · pp. 100–102
• 5,000 validation and 5,000 test images define M 2’s MS COCO evaluation split, following Karpathy and Fei-Fei (2014), with remaining images used for training.Evaluation uses BLEU, METEOR, ROUGE (Lin, 2004), CIDEr, and SPICE (Anderson et al., 2016).Source: §3.1.3.1 · p. 103
• 131.2 CIDEr with meshed connectivity exceeds reduced M 2 Transformer's 129.2 and the original Transformer's 121.8 (Cornia et al., 2020).The reduced model connects each encoder layer only to its corresponding decoder layer, whereas meshed connectivity accesses all encoder layers using sigmoid gating. Removing memory vectors reduces CIDEr by nearly one point in both reduced and full M 2 Transformer.Source: §3.1.3.1 · pp. 103–104
• M 2 Transformer leads compared models on BLEU-4, METEOR, and CIDEr on the single-model Karpathy test split (Cornia et al., 2020).It is also competitive with SGAE on BLEU-1 and ORT on SPICE.Source: Figure 3.8 · p. 104, §3.1.3.1 · p. 104
• Selected caption examples show M 2 Transformer producing more accurate descriptions and detecting more detailed image-region relationships than the original Transformer (Cornia et al., 2020).The examples include captions from both models alongside ground-truth descriptions.Source: §3.1.3.1 · p. 105, Figure 3.9 · p. 105
• M 2 Transformer learns multi-level image-region relationships and uses meshed decoding to exploit low- and high-level features during language generation (Cornia et al., 2020).Image encoding also exploits a priori knowledge from each encoder layer.Source: §3.1.3.1 · p. 105

3.2.1 Seeking objectivity

• Inception Score rewards class diversity and confident classifications, whereas FID compares distributions of real and generated Inception features.Salimans et al. (2016) introduced IS; Heusel et al. (2017) model FID features as Gaussians, addressing IS's omission of real data. Evaluation lacks an agreed metric, while FID aligns with human judgment and worsens with image distortions.Source: Inception Score (IS) · pp. 107–108, §3.2.1 · p. 107, Fréchet Inception Distance (FID) · p. 108
• Precision/recall distinguish realism from coverage, CLIP score assesses caption–image semantic similarity, and human evaluations assess photorealism and text alignment.Distribution-based precision/recall from Sajjadi et al. (2018), improved by Kynkäänniemi et al. (2019), expose trade-offs obscured by IS or FID. CLIP [CLIP2021] supplies the semantic matching measure.Source: Precision / Recall · pp. 108–109, CLIP score · p. 109, Human evaluations · p. 110

3.2.2.2 Conditioning on Text

• Reed et al. (2016c) condition GAN-INT-CLS's generator and discriminator on text embeddings, extending adversarial image generation to text-to-image synthesis.Following Goodfellow et al. (2014c), the generator attempts to fool a discriminator trained to distinguish real from generated images. Following Reed et al. (2016b), GoogLeNet and character-level CNN–RNN encoders learn aligned image/text representations before the image encoder is discarded.Source: §3.2.2.1 · p. 110, §3.2.2 · p. 110, §3.2.2.2 · p. 111
• GAN-CLS teaches discrimination of image realism and caption matching, while GAN-INT augments training with interpolated text embeddings.GAN-CLS labels real-image/right-text pairs true and both real-image/wrong-text and fake-image/right-text pairs false. GAN-INT exploits interpolations near the real-data manifold as synthetic conditioning examples that regularize training.Source: GAN-CLS · p. 113, GAN-INT · p. 113

Results

• GAN-INT-CLS performs best when combining conditioning and embedding interpolation, transferring pose and background and generating images from unseen classes.Experiments cover CUB birds and Oxford-102 flowers; the zero-shot examples are from Reed et al. (2016c). GAN-CLS also generalizes across MS-COCO domains, although its outputs are not always coherent.Source: Results · p. 113, Figure 3.18 · p. 114, Figure 3.19 · p. 115

3.2.2.3 Further GAN-like development

• StackGAN refines images through stacked generator–discriminator pairs, while AttnGAN adds word-level attention across three generation stages.Zhang et al. (2016a) condition Stage-II on text and the Stage-I image, omitting fresh noise to increase resolution from 64x64 to 256x256. Xu et al. (2017) use attention to guide local refinements and visualize which words receive attention.Source: StackGAN · p. 115, AttnGAN · p. 116
• DM-GAN uses dynamic memory to improve initial-image refinement, whereas DF-GAN replaces stacked generators with a single-stage backbone.Zhu et al. (2019) select text information through a memory-writing gate and merge memories with image features through a response gate. Tao et al. (2020) combine hinge loss and residual blocks with Matching-Aware Gradient Penalty for semantic consistency and regularization, plus One-Way Output for convergence.Source: AttnGAN · pp. 116–117

3.2.3 Dall-E 1

• 250 million text-image pairs supported a 12-billion-parameter Dall-E model with zero-shot capabilities comparable to earlier domain-specific models, as described by Ramesh et al. (2021b).Dall-E was not publicly available. Its introduction motivates variational autoencoders, which learn latent distributions rather than fixed representations and generate varied outputs by sampling before decoding.Source: §3.2.3 · p. 117, Variational Autoencoder (VAE) · p. 117

VQ-VAE / dVAE

• VQ-VAE uses learned discrete codebook vectors; DALL-E’s dVAE instead assigns probabilities across codebook entries.VQ-VAE replaces encoder outputs with their nearest codebook vectors under Euclidean distance. Training copies gradients from the decoder’s input to the encoder’s output to bypass discrete quantization.Source: VQ-VAE / dVAE · pp. 117–119

Dall-E system

• Over 90% of human evaluations preferred DALL-E over DF-GAN for realism and caption similarity on MS-COCO captions (Ramesh et al., 2021b).A dVAE compresses 256x256 images into 32x32 token grids; a transformer autoregressively models up to 256 byte-pair-encoded text tokens followed by 1024 image tokens. At inference, predicted image tokens are decoded by the pretrained dVAE.Source: Results · p. 119, Dall-E system · p. 119, Figure 3.24 · p. 120

3.2.4 GLIDE

• GLIDE generates more photorealistic zero-shot images, with human evaluators preferring its outputs over DALL-E for realism and caption similarity (Nichol et al., 2021b).Diffusion generates images by learning to reverse a Gaussian-noise Markov chain, following Sohl-Dickstein et al. (2015) and Ho et al. (2020a). Classifier-free inference guidance scales the difference between text-conditioned and unconditioned outputs and generally outperforms guidance using a separate CLIP classifier.Source: Diffusion models · p. 121, Results · pp. 122–123, GLIDE system · p. 122

3.2.5 Dall-E 2 / unCLIP

• unCLIP achieves unprecedented MS-COCO FID and greater diversity than GLIDE, with similar photorealism and caption similarity (Ramesh et al., 2022a).Its diffusion prior slightly outperforms an autoregressive prior at generating caption-conditioned CLIP image embeddings, which condition the image decoder alongside captions. The decoder uses classifier-free guidance and diffusion upsamplers from 64x64 through 256x256 to 1024x1024; the best reported results use guidance scale 1.25.Source: Dall-E 2 system · p. 124, Figure 3.30 · p. 125, Results · p. 125

Limitations

• Persistent limitations include composition, image text, complex prompts, counting, and negation; GLIDE also suffers slow sequential inference.Original DALL-E lacks photorealism; unCLIP’s authors hypothesize that CLIP embeddings worsen composition/text failures and that 64x64 decoding before upsampling loses complex-scene detail. Parti additionally blends features, omits or duplicates details, and mispositions objects, while GLIDE’s sequential denoising prevents straightforward parallelization.Source: Limitations · pp. 119, 123, 125, 129

3.2.6 Imagen & Parti

• Imagen and Parti significantly improve MS-COCO FID over prior work, delivering the section’s strongest reported results as of September 2022.Imagen uses frozen T5-XXL for richer language representations rather than a caption-trained encoder. Parti combines ViT-VQGAN (Yu et al., 2021) image tokenization with a Transformer-like autoregressive text-to-image sequence model.Source: §3.2.6 · pp. 127–128, Figure 3.35 · p. 128, Results · p. 128

3.2.7 Discussion

• Commercial incentives encourage closed models and training data, despite an active public-model community and emerging creative-production applications.Public examples include VQGAN-CLIP (Crowson et al., 2022), Midjourney, Latent Diffusion, and Stable Diffusion. Potential applications include texture synthesis, product design, virtual worlds, and production automation, aided by inpainting and outpainting.Source: Open- vs closed-source · p. 130, Open-Source Community · p. 130, Potential applications · pp. 130–131

Ethics / Conclusion

• Generated and edited images can enable harassment and disinformation, making responsible model sharing a significant challenge.Inpainting can add or erase objects in real scenes, strengthening this misuse risk. The section points to DALL-E 2’s risk discussion by Mishkin et al. (2022).Source: Ethics / Conclusion · p. 131
• Text-to-image models inherit biases and stereotypes from their training data, making broader diversity a requirement for commercialization.Models combine concepts from the data they have seen.Source: Ethics / Conclusion · p. 131
• Authorship of generated images remains unresolved, with potential claimants including prompt writers, model engineers, architecture designers, training-data owners, and the model itself.The passage presents resolving copyright questions as a prerequisite for fully enabling AI generation.Source: Ethics / Conclusion · p. 131
• Determining what constitutes a creative contribution deserving copyright protection requires extensive debate and, the authors hope, legal solutions.Source: Ethics / Conclusion · p. 131

3.3.1 Words In (Non-Symbolic) Contexts

• Visual grounding aims to improve semantic word embeddings by supplementing text-only distributional learning with perceptual context.The review follows concatenation, shared grounded spaces, and Transformers. Concrete applications such as machine translation remain largely outside its scope.Source: §3.3.1 · p. 133

3.3.2 Word-Embeddings: Survival-Kit

• Skipgram (Mikolov et al. (2013b)) and GloVe (Pennington et al. (2014)) produce context-free embeddings, whereas BERT (Devlin et al. (2018c)) represents tokens contextually.Skipgram predicts neighboring words, while GloVe models corpus-wide co-occurrences. BERT uses masked-token prediction and next-sentence prediction; model inputs are tokens, which need not be whole words.Source: §3.3.1 · p. 133, §3.3.2 · p. 134

3.3.3 The Beginning: Sequential Multimodal Embeddings

• Bruni et al. (2014) concatenate textual and visual semantic matrices, then use singular value decomposition to construct a shared multimodal space.Their visual features count clustered local descriptors; Kiela and Bottou (2014) instead concatenate CNN-derived features with textual embeddings. Silberer and Lapata (2014) combine unimodal denoising autoencoders through a bimodal autoencoder, adding a softmax layer for task-specific fine-tuning.Source: §3.3.3 · pp. 135–136

3.3.4 The Grounded Space

• Lazaridou et al. (2015) augment Skip-gram with a visual hinge loss and cross-modal mapping, enabling visual representations for words lacking training images.Grounded spaces address the limitations of one-to-one image associations, including missing images and difficulties depicting abstract words. During training, the visual loss is zero for words without associated images.Source: §3.3.4 · pp. 136–137
• Kiela et al. (2017) jointly train a linear grounding map and bidirectional LSTM to extend visual grounding from words to sentence representations.Different loss specifications support proposing alternative captions, predicting captions from images, or both, demonstrating reuse beyond pure language tasks.Source: §3.3.4 · p. 138

3.3.5 The Transformers Era

• Transformers combine self-attention, parallel input processing, and positional embeddings to capture dependencies throughout a sentence.Positional embeddings encode input positions, while self-attention provides access to earlier sequence information. The emphasis shifts from word similarity toward downstream tasks and transferable, task-agnostic representations requiring less labeled fine-tuning data.Source: §3.3.5 · pp. 139–140
• Flava (Singh et al. (2022)) and UniT (Hu and Singh (2021a)) illustrate Transformer architectures designed to handle linguistic, visual, and multimodal tasks.Flava combines separate image/text encoders with a multimodal encoder, using masked unimodal, contrastive, masked multimodal, and image-text matching pre-training objectives. UniT instead jointly trains separate encoders, a domain-agnostic decoder, and task-specific heads without multimodal pre-training, aiming to avoid knowledge loss from task-specific fine-tuning.Source: §3.3.5 · p. 140

3.3.5.1 Vokenization

• Around one third of tokens in Wiki103, English Wikipedia, and CNN/DM are visually grounded, exposing limited coverage for image-based supervision.Tan and Bansal (2020) propose associating tokens with images called vokens and adding softmax classification with negative log-likelihood loss. Grounding Ratio measures visually grounded token coverage, but an image set covering all tokens is unavailable.Source: §3.3.5.1 · pp. 140–141
• Vokenization retrieves a contextual image for every token from a finite image set, enabling visual supervision beyond directly grounded vocabulary.BERT and ResNeXt embeddings are projected into a shared, L2-normalized space, and caption-image pairs train relevance scoring with hinge loss. Normalized inner-product maximization permits nearest-neighbor retrieval; Revokenization transfers vokens between tokenizers.Source: §3.3.5.1 · pp. 142–144

3.3.5.2 One Step Further: The Power Of Imagination

• iACE provides model-agnostic generated-image supervision for pure language models or visually supervised models (Lu et al. (2022)).VQGAN (Esser et al. (2021)) generates images, while CLIP (Radford et al. (2021a)) assesses textual alignment and serves as the cross-modal encoder. Masked-language and Voken-Classification pre-training precede imagination-augmented fine-tuning on GLUE and SWAG.Source: §3.3.5.2 · pp. 144–146, Figure 3.44 · p. 145

3.3.6.1 Evaluation In The Pre-Transformers Era

• Pre-Transformer visual grounding improves representations of concrete concepts over language-only baselines, but offers little benefit for abstract concepts.Intrinsic evaluation correlates embedding cosine similarities with human word-similarity judgments. These judgments depend on observers’ experience, culture, and preferences, motivating the shift toward downstream-task evaluation.Source: §3.3.6.1 · pp. 146–147, §3.3.6.2 · p. 147

3.3.6.2 Evaluation In The Post-Transformers Era

• Voken-Classification pre-training can outperform pure language baselines, whereas universal Transformer models do not surpass BERT or RoBERTa (Tan and Bansal (2020)).Intrinsic evaluation likewise finds universal vision-language models generally behind domain-specific models, with Vokenization the principal competitor (Pezzelle et al. (2021)).Source: §3.3.6.2 · pp. 148–149, Figure 3.48 · p. 149, Figure 3.49 · p. 150
• Up to 5% of training data suffices for imagination-supervised models to outperform Voken-supervised models in the reported few-shot GLUE comparison (Lu et al. (2022)).Costly collection of paired images, texts, and labeled examples motivates evaluating performance with limited training data.Source: §3.3.6.2 · pp. 150–151, Figure 3.51 · p. 151

3.3.7 The End Of This Story

• Broader perceptual inputs, including video and speech, remain an open direction beyond static-image grounding.Task-agnostic pre-training transfers representations to downstream tasks to reduce costly labeling requirements, making few-shot generalization increasingly important. Applications include dialogue response generation (Sun et al. (2021)) and translation, where images help with generic or ambiguous textual context (Ive et al. (2019)).Source: §3.3.7 · pp. 151–152

3.3.8 Appendix: Selected Models - Summary

• The appendix introduces a summary table of selected language models augmented with visual components.The supplied passage does not include the table’s entries.Source: §3.3.8 · p. 152
• Language models and visual features are considered alongside their respective pretraining data.Source: §3.3.8 · p. 152
• Fusion strategy is a separate comparison dimension for the two modalities.Source: §3.3.8 · p. 152
• Evaluation coverage includes benchmarks and baselines.Source: §3.3.8 · p. 152

3.4.2.1 Web-scale data; 3.4.2.3 Foundation models and zero-shooting

• Web-scale image–text supervision avoids manual labeling but requires curation, reproduces social biases, and chiefly compensates for poor data efficiency.Foundation models adapt to varied downstream tasks; zero-shot evaluation requires no target-dataset fine-tuning and avoids overfitting introduced by that adaptation. Classification selects the class text most similar to an image embedding, while retrieval selects images most similar to a text prompt.Source: §3.4.2.1 · p. 154, §3.4.2.3 · pp. 157–158

3.4.2.2 Contrastive objective

• Fourfold higher data efficiency follows replacing bag-of-words prediction with CLIP’s contrastive objective (Radford et al., 2021a).Joint image and text encoders maximize similarities for matching pairs and minimize mismatches; exact-caption prediction trains three times slower than bag-of-words prediction in data-efficiency terms. Contrastive models select among available options rather than freely generating text or images, requiring additional network components for generation.Source: §3.4.2.2 · pp. 154–156, Figure 3.54 · p. 157

3.4.3.1 CLIP

• Zero-shot CLIP remains more accurate under distribution shifts than an ImageNet-trained ResNet101 matched for original ImageNet accuracy (Radford et al., 2021a).Its best image encoder is ViT-L/14, subsequently trained at higher resolution; contextualized class labels and prompt ensembles improve zero-shot classification. However, Shen et al. (2021) find ViT-B weaker than ResNet backbones for vision–language tasks, potentially because its pooling impairs localization, while CLIP’s private training data prevent independent robustness verification.Source: Architecture · pp. 159–160, Figure 3.56 · p. 160, Robustness · p. 160

3.4.3.2 ALIGN; Connecting image and text representations

• 1.8 billion image–text pairs let ALIGN prioritize minimally filtered scale over extensive curation, accepting increased noise.ALIGN combines BERT-Large and EfficientNet-L2 encoders trained jointly from scratch, replicating CLIP’s performance with substantially less curation. Its aligned representation space also supports combining image and text embeddings arithmetically for image retrieval.Source: §3.4.3.2 · p. 161, Architecture · p. 161, Connecting image and text… · p. 162

3.4.3.3 Florence

• Florence extends a pretrained core with modules targeting fine-grained object detection, action recognition, and multimodal tasks.Its CoSwin image encoder and CLIP-like text Transformer comprise 893 million parameters trained jointly from scratch on 900 million image–text pairs. Its image-label-description contrastive objective treats every image–text pair sharing a label as positive.Source: §3.4.3.3 · p. 162, Architecture · pp. 162–163

3.4.4 Performance comparison

• 7.3pp separates Florence from both CLIP and ALIGN in zero-shot accuracy on original ImageNet.Florence reports no results on the other three ImageNet variants, where CLIP and ALIGN have no clear winner. Zero-shot retrieval comparisons on Flickr30K and MSCOCO rank Florence first, ALIGN second, and CLIP third.Source: Figure 3.61 · p. 164, §3.4.4 · p. 164, Figure 3.62 · p. 164
• Florence leads overall, outperforming CLIP in nearly all reported zero-shot and fine-tuned evaluations.ALIGN reports results on four of twelve datasets and wins half those comparisons. None of the training datasets were public, so evaluation on entirely unseen data cannot be guaranteed.Source: §3.4.4 · p. 164, Figure 3.63 · p. 165

3.4.5 Resources

• CLIP has accessible pretrained models and retrieval tools, whereas the review reports no open-sourced resources accompanying ALIGN or Florence.rclip enables personal image retrieval, and DALL·E 2 uses CLIP for image embeddings (Ramesh et al., 2022b). LAION-400M seeks to replicate CLIP’s training dataset, filtering collected image–text pairs by their CLIP cosine similarity.Source: §3.4.5 · p. 165

3.5 Models for both modalities

• Multimodal modeling seeks contextual understanding and unified frameworks, but faces expensive labeling, limited data and training time, and modality-specific input transformations.Self-supervision and automatically scraped web datasets address data acquisition constraints, yet models with billions of parameters can strain even web-scale resources.Source: §3.5 · pp. 165–166

3.5.1 Data2vec

• 800 epochs of data2vec training outperform MAE and MaskFeat, both trained for 1600 epochs.ImageNet-1K top1-accuracy also exceeds MoCov3 (Chen et al., 2021) and DINO (Caron et al., 2021). The section reports state-of-the-art vision performance among similar self-supervised models.Source: Figure 3.67 · p. 170, §3.5.1 · pp. 170–171
• Data2vec improves average GLUE performance over retrained RoBERTa, but gains vary by task and appear largely driven by CoLA.Averaging multiple teacher layers improves all three domains, with vision gaining nothing beyond three layers and speech reaching peak performance around six. Feedforward-layer outputs work best as prediction targets.Source: Figure 3.68 · p. 171, §3.5.1 · pp. 171–172
• Data2vec unifies self-supervised text, speech, and image learning through contextualized teacher–student targets, but does not learn cross-modal interactions.An exponential-moving-average teacher sees complete inputs; the masked student regresses normalized averages of its top K layers using smooth L1 loss. Encoding and masking remain modality-specific, including masking 60% of image patches.Source: §3.5.1 · pp. 167, 168, 169, 172

3.5.2 Vision-and-Language Bert (VilBert)

• VilBert connects separate vision and language transformer streams through co-attention, exchanging keys and values to condition each modality on the other.Faster R-CNN supplies spatially encoded image-region features, while separate streams permit modality-specific processing without forcing shared visual/text tokens. Pre-training combines masked multimodal modeling and image–caption alignment prediction on Conceptual Captions.Source: §3.5.2 · pp. 173, 174, 176
• Approximately 15% lower VQA performance follows word shuffling, although VilBert attention maps barely change.Nouns dominate attention, prepositions sometimes help spatial relations, and object-region proposals can bottleneck fine-grained maps. In the illustrated example, 72 or 108 proposals yield correct answers despite attention that differs substantially from human attention.Source: §3.5.2 · pp. 174–175, Figure 3.71 · p. 175
• VilBert’s dual-stream architecture outperforms a single stream, with further gains from pre-training, fine-tuning, and larger datasets (Lu et al., 2019b).Optimal depth is task-dependent: VQA and image retrieval peak at six layers, zero-shot image retrieval benefits from deeper networks, and VCR and RefCOCO+ favor shallower models.Source: Figure 3.72 · p. 176, §3.5.2 · p. 176

3.5.3 Flamingo

• Flamingo accepts interleaved visual–text prompts for chatting, image description, and video tasks, adapting through only a few examples.Text attends directly only to the preceding image, enabling arbitrary image counts beyond those used in training. Seven benchmarks guide development and eleven estimate unbiased performance, with evaluation guarding against uncounted tuning prompts highlighted by Perez et al. (2021b).Source: §3.5.3 · pp. 177, 179, 182
• Flamingo connects frozen pretrained vision and language models through a learnable perceiver-resampler and gated cross-attention layers.The resampler produces a fixed set of visual tokens, while zero-initialized tanh gating preserves language-model stability at initialization. Hardware considerations led to inserting cross-attention every seventh language-model layer.Source: §3.5.3 · pp. 178, 181, 182
• 68.4 fell to 46.9 without web-scraped multimodal data, 56.5 without aligned captions and images, and 59.7 without gradient accumulation across datasets.Training combines webpage image–text sequences with ALIGN, LTIP, and VTP to provide heterogeneous data. Optimization minimizes dataset-weighted expected negative log likelihood; tuning weights and accumulating gradients before updates were important.Source: §3.5.3 · pp. 179–180
• Around three to five score points favored the perceiver over transformer or MLP alternatives, while medium-sized perceivers delivered improvements around two to three points.Learned latent queries compress flattened visual features, reducing subsequent cross-attention complexity, particularly for video. Latent-query keys and values are concatenated with visual-feature keys and values; oversized perceivers can destabilize training with large frozen language models.Source: §3.5.3 · pp. 180–181
• Six tasks saw Flamingo exceed the general state of the art, while it outperformed all previous few-shot models without fine-tuning.Larger models and more shots improved performance. Additional fine-tuning substantially improved results further, but required expensive training and hyperparameter tuning.Source: §3.5.3 · pp. 182–183, Figure 3.78 · p. 183, Figure 3.79 · p. 184
• Contrastive models outperform Flamingo on classification, although its few-shot learning and open-ended generation offer broader application possibilities.The authors hypothesize that text–image retrieval training is particularly important for classification.Source: §3.5.3 · pp. 183–184

3.5.4 Discussion

• 80B parameters make Flamingo larger than data2vec and VilBert; the discussion associates larger models with stronger performance and greater generality.Resource requirements and restricted releases limit access: the passage contrasts VilBert’s public codebase with generally inaccessible data2vec and Flamingo models. Sparse activation in Pathway models reduces energy consumption below 10% of that expected for similar dense models.Source: §3.5.4 · pp. 184–185
• Reusing frozen pretrained models and enabling communication between them offers efficient adaptation while protecting existing capabilities; fine-tuning may still deliver better absolute results.The discussion warns about model collapse and stresses heterogeneous data, dataset-weighted optimization, and unbiased few-shot evaluation. Socratic models illustrate complementary model knowledge through multimodal dialogue and robot perception.Source: §3.5.4 · p. 185

4.1 Including Further Modalities

• A challenge-based taxonomy organizes multimodal learning beyond text and images without enumerating every possible combination of modalities.Video introduces temporal information, while audio, speech, and other sensor signals broaden the inputs considered. The taxonomy aims to make unfamiliar modalities approachable through shared learning challenges.Source: §4.1 · pp. 188–189

4.1.1.1 Multimodal Representation Learning

• Joint representations combine modalities in one representation space, whereas coordinated representations align separate modality-specific spaces through a learning objective.Both approaches assume meaningful semantic relationships between modalities; coordinated learning brings related samples closer while separating dissimilar ones.Source: §4.1.1.1 · pp. 191–192

4.1.1.2 Multimodal Alignment

• Multimodal alignment makes separate embedding spaces comparable, commonly using contrastive learning to attract semantically equivalent cross-modal samples and repel dissimilar ones.This approach assumes a dataset of corresponding multimodal samples.Source: §4.1.1.2 · p. 192

4.1.1.3 Multimodal Fusion

• Multimodal fusion learns joint representations by merging information across modalities early, midway, late, or through hybrid arrangements within a model.For Transformer-based models (Vaswani et al., 2017f), fusion concerns where inputs begin attending to one another across modalities.Source: §4.1.1.3 · p. 193

4.1.1.4 Multimodal Translation

• Multimodal translation retrieves existing matches across aligned representation spaces or generates new outputs by decoding learned representations.Generation requires a sufficiently smooth distribution for meaningful interpolation, often demanding careful regularization and challenging evaluation. Diffusion models (Ho et al., 2020b) can produce photorealistic outputs while using pretrained CLIP (Radford et al., 2021c) backbones.Source: §4.1.1.4 · pp. 193–194

4.1.2 Current Research Trends: Generalized Self-Supervised Multimodal Perception

• Generalized multimodal perception requires both suitable architectures and training paradigms; the authors propose combining modality-agnostic models with generalized non-contrastive self-supervised learning.Their proposed research covers unimodal and multimodal learning. Desired architectural properties include input-agnostic processing, fusion, locality, compositionality, and flexible outputs.Source: §4.1 · p. 189, §4.1.2 · p. 195, §4.1.2.1 · p. 195

4.1.2.1 General Multimodal Architectures

• NÜWA represents inputs in a common three-dimensional format and applies local attention to preserve spatial-temporal structure while reducing attention costs.Its encoder-decoder supports modality-agnostic processing and flexible outputs when inputs use the required format. A hierarchical extension respecting compositionality was not studied in the original paper.Source: §4.1.2.1 · pp. 195–197
• More than 300x compression lets Perceiver IO [Jaegle et al. (2021b)][@ jaegle2021perceiver] process large multimodal inputs through latent-array cross-attention and decode arbitrary multimodal outputs.A smaller latent array supplies queries while the input supplies keys and values. Concatenated multimodal inputs allow the model to learn joint representations.Source: §4.1.2.1 · p. 197
• Hierarchical Perceiver merges group-specific latent outputs across encoder levels, improving locality, compositionality, and computational costs relative to Perceiver IO.Each input group cross-attends to its own latent array, followed by self-attention and MLP blocks; merged outputs become input to the next hierarchical block.Source: §4.1.2.1 · pp. 197–198

4.1.2.2 Multimodal Training Paradigms

• BYOL predicts teacher latents from augmented inputs; data2vec extends this approach across modalities using modality-specific masking and encoders.The teacher follows an exponential moving average of student weights. Latent prediction uses regression rather than data-space reconstruction or hard-negative mining, aiming to improve contextual representations (Baevski et al., 2022).Source: §4.1.2.2 · pp. 199–200
• VATT (Akbari et al., 2021) learns hierarchical video–audio–text embedding spaces through contrastive objectives that encode differences in modalities’ semantic granularity.Its framework introduces Multiple-Instance-Learning-NCE and combines contrastive losses at different levels, first aligning video with audio and then text with the combined representation. Extension to further modalities requires corresponding modality tuples in the dataset.Source: §4.1.2.2 · p. 200
• MultiMAE (Bachmann et al., 2022) learns joint representations from masked RGB, depth, and semantic-map patches while supporting modality-specific reconstruction tasks.A multimodal transformer and global token integrate information across modalities; modality-specific decoders use cross-attention to reconstruct their inputs. Their aggregate reconstruction loss trains the model.Source: §4.1.2.2 · pp. 200–201
• The proposed non-contrastive framework extends latent prediction to arbitrary modalities, targeting joint and coordinated representations using a Perceiver-format input and one masking strategy.A Perceiver backbone handles large arrays, while contiguous masking avoids trivial prediction of correlated signals. Cross-modal training treats modalities as semantic augmentations fed to student and teacher, avoiding modality-specific models, masking schemes, and contrastive hard negatives.Source: §4.1.2.2 · pp. 202–203

4.2.2 Taxonomy: Structured vs. Unstructured Data

• The distinction between structured and unstructured data depends on dimensionality and individual-feature interpretability, with high-dimensional biomarkers illustrating blurred boundaries.Traditional analyses often transform unstructured inputs using expert knowledge or PCA, whereas deep learning extracts features directly. The chapter motivates joint modeling because discarding either available data type can lose useful information and potential performance improvements.Source: §4.2.1 · pp. 203–204, §4.2.2 · pp. 204–205

4.2.3 Fusion Strategies

• Early, joint, and late fusion combine input features, intermediate representations, and independently trained predictions, respectively, following the taxonomy of Huang et al. (2020).Joint fusion enables end-to-end feature learning through backpropagation, with neural heads capturing interactions or statistical heads preserving interpretability. Late fusion aggregates predictions through approaches such as averaging or voting but is not investigated further here.Source: §4.2.3 · pp. 205–206

4.2.4.2 Traditional Survival Analysis (CPH Model)

• The Cox Proportional Hazard model assumes linear covariate effects on risk, yielding interpretable proportional, multiplicative hazard effects but limiting flexibility.Survival analysis studies time until an event occurs. Multimodal survival research extends traditional clinical-record inputs with CT images and gene-expression data.Source: §4.2.4.1 · pp. 206–207, §4.2.4.2 · p. 207

4.2.4.3 Multimodal DL Survival Analysis

• Yao et al. (2017) align pathological-image and molecular representations in DeepCorrSurv before survival fine-tuning, targeting modality discrepancy and noise in small samples.DeepSurv replaces linear effects with neural modeling of structured covariates (Katzman et al. (2018)); DeepConvSurv learns image features end-to-end, outperforming handcrafted features (Zhu et al. (2016)). Tong et al. (2018) concatenate separately learned autoencoder features in ConcatAE to capture complementary information, whereas CrossAE reconstructs both modalities from either input to capture common information; applications were limited to multi-omics rather than tabular EHR data.Source: §4.2.4.3 · pp. 207–208
• Cheerla and Gevaert (2019) combine similarity loss with multimodal dropout to learn modality-invariant survival representations and accommodate missing modalities.Their end-to-end model covers 20 cancer types and four modalities, dropping entire modalities during training while survival loss guides representation learning. T-SNE embeddings cluster patients by cancer type without training on that variable, supporting patient profiling.Source: §4.2.4.3 · p. 208, Figure 4.11 · p. 209
• Structured clinical plus mRNA data performed best in MultiSurv, outperforming single-modality models; all six modalities failed to beat clinical-only CPH (Vale-Silva and Rohr (2021)).Across 33 TCGA cancer types, adding modalities slightly reduced performance, with substantial cancer-specific variability and sometimes only about 20 training observations. Small samples and repeated testing of modality combinations raise robustness concerns.Source: §4.2.4.3 · pp. 208–209
• Pölsterl et al. (2019) and Kopper et al. (2022) combine structured and visual survival inputs with additive linear heads that preserve variable-level interpretability.Following Cheng et al. (2016), linear and neural components support memorization and generalization; Pölsterl et al. (2019) use CPH, whereas Kopper et al. (2022) use PAMM to relax linearity and proportionality through piecewise-constant hazards. Overlapping inputs create identifiability problems because changes in linear coefficients can be offset by neural-network adjustments without changing predictions.Source: §4.2.4.3 · pp. 210–211

4.2.4.4 Multimodal DL in Other Scientific Fields

• Rügamer et al. (2020) introduce SSDDR to learn distributional parameters through linear, generalized additive, and neural components, incorporating uncertainty quantification.An orthogonalization cell separates deep representations from overlapping structured effects, enabling backpropagation while preserving interpretability. Using two-stage orthogonalization for London housing attributes and imagery, Law et al. (2019) improve on single-modality predictions; structured data remains strongest alone, and fully nonlinear heads offer only slight gains over interpretable additive heads.Source: §4.2.4.4 · pp. 211–213
• About 516 million buildings across roughly 64% of Africa were mapped using CNNs by Sirko et al. (2021), demonstrating a large-scale remote-sensing application.Jean et al. (2016) predict African poverty from satellite images, while Gebru et al. (2017) estimate US socioeconomic attributes from Street View images, offering low-cost alternatives to costly, irregular surveys. You et al. (2017) predict US corn yield using Deep Gaussian Processes; these satellite-based approaches address scarce labels and could be combined with structured data for economic and financial applications.Source: §4.2.4.4 · pp. 213–214

4.2.5 Conclusion and Outlook

• Multimodal models almost always outperformed single-modality models when compared, while structured data usually provided the strongest single-modality predictions.End-to-end learning frequently surpassed independently learned or expert-derived features. Interpretable models could retain high predictive performance, with unstructured data supplying supplementary information.Source: §4.2.5 · p. 214
• Incomplete single-modality benchmarking, extensive model and modality selection, small datasets, and possible publication bias limit confidence in reported multimodal improvements.Apparent gains may hold only for particular configurations, especially in small biomedical datasets. Future progress likely requires sufficiently large samples of task-relevant structured and unstructured modalities, rather than simply adding modalities.Source: §4.2.5 · pp. 214–215

4.3.1 Prerequisites

• Mixture-of-experts reduces computation through gated sparse activation, while evolutionary algorithms optimize discrete search spaces through fitness-based selection, mutation, and recombination.Expert balancing prevents routing collapse. Evolutionary search balances exploration and exploitation, repeatedly retaining fitter individuals until a stopping criterion is met.Source: §4.3.1.2 · p. 216, §4.3.1.3 · p. 217
• Multipurpose models combine modalities with simultaneous multitask training, seeking transferable knowledge and better generalization, although conflicting task demands can reduce performance.Multitask learning (Caruana (1997), Crawshaw (2020)) exploits patterns shared across tasks rather than learning each task independently.Source: §4.3.1.1 · p. 216, §4.3.1.4 · p. 217

4.3.2.1 MultiModel

• MultiModel connects modality nets through an encoder, I/O mixer, and decoder, supporting text and classification outputs with low-resource transfer benefits.Its extensible pre-transformer architecture uses a unified representation space, convolutional/MoE encoding, and cross-attention. Joint training remained below state of the art and did not significantly outperform single-task specialists, except on a low-resource task.Source: §4.3.2.1 · pp. 217–218

4.3.2.2 Unified Transformer (UniT)

• UniT (Hu and Singh, 2021b) combines modality-specific encoders with a non-autoregressive decoder, outperforming its single-task version on multimodal tasks but losing on unimodal tasks.Task tokens, decoder query vectors, and task-specific heads specify the requested output; additional encoders support extension to new modalities. Results remained below, but comparable to, state-of-the-art models.Source: §4.3.2.2 · pp. 218–219

4.3.2.3 OFA - Sequentialization is All You Need

• OFA surpassed state-of-the-art models on image captioning, visual question answering, visual entailment, and visual grounding while sequentializing inputs and outputs.Shared-space embeddings feed an encoder–decoder transformer; generated image tokens and bounding-box coordinates require downstream decoding or visualization. Unimodal results did not surpass state of the art, and few-shot transfer to unseen tasks lacked specialist-baseline evaluation.Source: §4.3.2.3 · pp. 220–221

4.3.2.4 Gato - A Generalist Decoder

• Gato uses a decoder-only transformer across modalities, usually outperforming specialist Gato variants but trailing specialist reinforcement-learning agents.Tokenized observations and actions support autoregressive prediction; unseen tasks required fine-tuning because input-length restrictions prevented few-shot learning, with mixed gains. Robotics behavior was comparable to a state-of-the-art baseline, while image captioning and dialogue capabilities were reported without further exploration.Source: §4.3.2.4 · pp. 221–222

4.3.2.5 Comparison

• OFA alone achieved state-of-the-art results among the reviewed models, while differing task and modality coverage makes direct comparison difficult.MultiModel, UniT, and Gato were broadly comparable to specialists. Transfer evidence remained qualified by low-resource testing, missing specialist baselines, and task-dependent outcomes.Source: §4.3.2.5 · pp. 223–224

4.3.3.1 Pathways Proposal

• Pathways proposes sparse paths through specialist neural networks to reuse learned capabilities without activating every network for each task.Routing tasks to suitable experts is expected to improve generalization, with multimodality considered a potential extension. This addresses inefficient retraining from scratch and the resource demands of increasingly large dense networks.Source: §4.3.3 · p. 224, §4.3.3.1 · p. 225

4.3.3.1 Pathways Proposal · 4.3.4 Conclusion Pathways

• Pathways proposes sparse activation to reduce inference time; the review recommends combining sparse multimodal processing with knowledge-preserving multitask learning.Its data-center framework aims to reduce computation overhead by accessing only required model chunks. Proposed research includes scaling PathNet and muNet beyond few tasks, intelligently routing their task-specific models, and combining LIMoE sparsity with OFA/Gato encoding.Source: §4.3.3.1 · p. 226, §4.3.4 · p. 231

4.3.3.2 PathNet

• PathNet (Fernando et al., 2017) improved second-task training time and prediction quality over fine-tuning and learning from scratch without catastrophic forgetting.An evolutionary algorithm trains competing paths, replaces losing paths with mutated winners, and freezes learned paths before subsequent tasks. New task-specific heads and reinitialized remaining parameters allow further learning while reusing frozen knowledge.Source: §4.3.3.2 · pp. 226–227

4.3.3.3 LIMoE

• LIMoE-H/14 outperformed CLIP (Radford et al., 2021a) with minimal additional parameters; similarly performing models used at least twice the parameters per forward pass.LIMoE uses one contrastively trained text–vision transformer, with gating that routes tokens to K feed-forward experts. Modality-specific local entropy losses encourage selective routing, while global entropy losses balance expert use.Source: §4.3.3.3 · pp. 227, 229

4.3.3.4 muNet (Multitask Network)

• muNet outperformed a fine-tuned ViT across classification tasks while using significantly fewer parameters and preserving previously learned knowledge.Evolutionary search balances exploration and exploitation when selecting candidates for mutation, training, and evaluation, ultimately retaining the best models for each task. Mutations clone, insert, or remove layers or change hyperparameters, with unchanged shared layers remaining frozen.Source: §4.3.3.4 · pp. 229–231

4.3.5 Discussion

• Multipurpose models can outperform specialists, but Gato’s weaker ATARI Boxing performance indicates that relationships between tasks still require investigation.Model size limits accessibility, potentially requiring API access, while existing metrics inadequately cover multitask and multimodal evaluation. Bias across multiple datasets and the environmental impact of training larger models remain concerns.Source: §4.3.5 · p. 232

4.4.1 Historical Overview

• DeepDream, style transfer, and StyleGAN preceded text-guided artist–critic generation, which enabled control over otherwise difficult-to-specify outputs.Mordvintsev (2015) visualized CNN activations with DeepDream; Gatys et al. (2016) transferred styles; Karras et al. (2019) generated realistic synthetic faces. CLIP (Radford et al., 2021b) guided BigGAN, StyleCLIP, VQGAN, and diffusion generators, while GLIDE supports CLIP or classifier-free guidance.Source: §4.4.1 · pp. 233, 235

4.4.2 How to use these models?

• Public notebooks and interfaces simplify text-to-image generation, but prompt wording, abstraction, model size, and computational limits affect outputs.Across CLIP + BigGAN, CLIP + VQGAN, CLIP + Guided Diffusion, and GLIDE, the landscape prompt produced relatively realistic images, whereas the abstract panda prompt yielded more varied outputs. Released GLIDE had 300 million parameters versus about 3.5 billion unreleased; computational limits also reduced some notebooks’ output resolution.Source: §4.4.2 · pp. 235–236, Figure 4.25 · p. 236, Figure 4.26 · p. 237

4.4.3 Different tasks and modalities

• Text-to-image models can start from noise or a real image, supporting both new image generation and transformations of existing imagery.The section illustrates an image-based starting point with the LMU seal.Source: §4.4.3 · p. 236
• VQGAN initialized from the LMU seal rather than noise produced sunflower imagery and a possible facial fragment under the prompt “in style of Van Gogh”.The accompanying video records how the image develops during fitting.Source: §4.4.3 · p. 237
• GLIDE supports text-guided image generation, inpainting, and sketch-based editing while matching surrounding image style.Other applications include accompanying custom videos with suitable audio and potentially creating sculptures with 3D printers.Source: §4.4.3 · p. 237

4.4.4 Discussion and prospects

• Human artists commonly select generated images and refine prompts, while no direct loss function measures aesthetic appeal.The section treats aesthetic judgment as subjective and argues that improving models reduce the amount of human intervention required.Source: §4.4.4 · p. 238
• Realistic generated content risks facilitating plagiarism and misleading information, creating tension between restricting model access and enabling research.The section describes some organizations withholding models or downstream models in response to misuse concerns, while emphasizing researchers’ need for access.Source: §4.4.4 · p. 238
• Generative models inherit input-data biases, potentially affecting which people appear in generated images.Esser et al. (2020) report that CLIP text embeddings associate a human being more with a man than a woman, potentially favoring male outputs.Source: §4.4.4 · p. 238
• DALLE-2 outperforms DALLE-1 (Ramesh et al., 2022a), illustrating continuing improvements in generative-art methods.The supplied passage gives no benchmark values or evaluation details for this comparison.Source: §4.4.4 · p. 239

5 Conclusion

• Text-to-image systems such as DALL-E have attracted public and artistic interest, with potential commercial applications in film and gaming but unresolved copyright concerns.Researchers and online communities have both explored these models and their outputs.Source: §5 · p. 241
• Realistic generated images can enable misinformation and public-opinion manipulation, motivating calls for automated fabrication-detection tools.Distinguishing fabricated images from real ones by visual inspection alone can be difficult.Source: §5 · p. 241
• Bias requires further investigation, while opaque decision-making can limit multimodal models’ social acceptance and usability.In medical applications, doctors and patients need the reasoning and certainty behind predictions, not just the predicted outcome.Source: §5 · p. 241
• About 175 billion parameters in GPT-3 (Brown et al., 2020), versus about 1.5 billion in GPT-2 (Radford et al., 2019a), illustrate increasing architectural scale.Increasing parameter counts often improves performance, but training and storing those parameters demands substantial time, computational power, and storage.Source: §5 · pp. 241–242
• 1,507 kWh of energy consumption and 1,438 lbs of CO2 were estimated for one BERT training run.GPT-2 training took 168 hours on 32 TPUv3 chips, with estimated cloud compute costs of $12,902–$43,008. Development and adaptation often require multiple fitting and tuning phases.Source: §5 · p. 242
• Concentrated computational and financial resources create unequal academic–industry access, while withheld models obstruct independent reproduction and raise concerns about monopoly and abuse.Companies may retain their best models as products and intellectual property rather than publishing them.Source: §5 · p. 242

6 Epilogue

• Limited project time and rapidly evolving research prevent exhaustive coverage, although students incorporated some research published during the seminar.Chapter topics were selected in February 2022, before some architectures, improvements, and ideas had been published.Source: §6 · p. 243

6.1 New influential architectures

• Stable Diffusion enables photorealistic image generation and is available open-source, with a demonstration accessible through Hugging Face.The model was introduced in “High-Resolution Image Synthesis with Latent Diffusion Models” by researchers from Björn Ommer’s group at LMU.Source: §6.1 · p. 243

6.2 Creating videos

• Google Research’s Imagen architecture was extended to generate videos from natural-language input, illustrating research beyond text-to-image generation.The epilogue presents this as one example among many efforts in this direction.Source: §6.2 · p. 244