PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Foundation Models: From Specialists to General-Purpose Assistants

Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao

arXiv:2309.10020v1cs.CVcs.CL

TL;DR

Multimodal foundation models are evolving from specialized vision and vision-language systems toward general-purpose assistants, but their taxonomy and open research directions remain fragmented. This survey organizes the field into five topics across specific-purpose pretraining and exploratory assistant-oriented approaches, highlighting a transition toward unified understanding, generation, and tool use.

  • Problem

    The field lacks a consolidated account of how multimodal foundation models progress from specialized capabilities toward general-purpose assistants with instruction following and alignment.

    Source: Introduction · p. 5, §5.1.3 · p. 64

  • Method

    The paper surveys five topics spanning visual understanding, text-to-image generation, unified vision models, end-to-end multimodal LLMs, and multimodal tool chaining.

    Source: Introduction · p. 5, §7.1 · p. 89

  • Results

    The survey organizes multimodal foundation models into specific-purpose pretraining and exploratory general-purpose assistant approaches, tracing the field’s transition between them.

    Source: §7.1 · p. 89

  • Takeaways & Limitations

    The synthesis provides a structured view of established methods and emerging directions for multimodal foundation models spanning understanding, generation, and assistance.

    Source: §7.1 · p. 89

    A fully integrated image-text-input, image-text-output foundation model balancing multimodal understanding and generation remains an open direction for future exploration.

    Source: §3.6 · p. 41

Abstract

from arXiv · show

This paper presents a comprehensive survey of the taxonomy and evolution of multimodal foundation models that demonstrate vision and vision-language capabilities, focusing on the transition from specialist models to general-purpose assistants. The research landscape encompasses five core topics, categorized into two classes. (i) We start with a survey of well-established research areas: multimodal foundation models pre-trained for specific purposes, including two topics -- methods of learning vision backbones for visual understanding and text-to-image generation. (ii) Then, we present recent advances in exploratory, open research areas: multimodal foundation models that aim to play the role of general-purpose assistants, including three topics -- unified vision models inspired by large language models (LLMs), end-to-end training of multimodal LLMs, and chaining multimodal tools with LLMs. The target audiences of the paper are researchers, graduate students, and professionals in computer vision and vision-language multimodal communities who are eager to learn the basics and recent advances in multimodal foundation models.

Introduction

This survey traces multimodal foundation models from specific-purpose visual understanding and generation toward unified, LLM-integrated, general-purpose visual assistants. It reviews established methods and emerging approaches for researchers, students, and professionals in vision and vision-language communities.

Introduction

  • The survey frames multimodal foundation models as progressing from specific-purpose pre-trained vision models toward general-purpose assistants that follow human intents across diverse computer vision tasks.Its taxonomy covers visual understanding, visual generation, unified vision modeling, end-to-end training with LLMs, and chaining multimodal tools with LLMs.Source: Introduction · p. 6, §1.2 · p. 9

1.1 What are Multimodal Foundation Models?

  • Multimodal foundation models emphasize vision and vision-language capabilities, spanning visual representations for image-, region-, and pixel-level understanding, multimodal fusion, and text-conditioned visual generation.The survey also discusses human-aligned generation, including controllability, prompt adherence, editing, and concept customization.Source: §1.1 · pp. 7–8

1.2 Definition and Transition from Specialists to General-Purpose Assistants

  • General-purpose visual assistants combine unified architectures with instruction following, pursued through unified vision models, end-to-end multimodal LLM training, and LLM-based tool chaining.Unified models address understanding and generation without explicitly using pre-trained LLMs, whereas the other approaches explicitly involve LLMs.Source: §1.1 · pp. 8–9, §1.2 · p. 9

1.3 Who Should Read this Paper?

  • The paper targets graduate students, researchers, and professionals seeking foundational knowledge, recent advances, and perspectives on multimodal foundation model trends.Its seven chapters progress from visual backbone learning and generation to unified models, end-to-end multimodal LLMs, tool chaining, and research trends.Source: §1.3 · p. 9

1.4 Related Materials: Slide Decks and Pre-recorded Talks

  • The survey extends the CVPR 2023 tutorial with recent field advances and accompanying slide decks and pre-recorded talks organized by chapter topic.Materials cover visual and vision-language pre-training, text-to-image alignment, vision interfaces, large multimodal models, and multimodal agents.Source: §1.4 · p. 11

Visual Understanding … 2.3 Contrastive Language-Image Pre-training

The chapter surveys vision foundation-model pre-training across label, language, and image-only supervision, emphasizing CLIP’s scalable image-text alignment, zero-shot capabilities, and extensions toward richer supervision, modalities, and multimodal fusion.

Visual Understanding

  • Vision backbones provide general-purpose image representations foundational to image-, region-, and pixel-level computer vision tasks.The survey organizes representation learning by label supervision, language supervision, and image-only self-supervision, then discusses multimodal fusion and finer-grained understanding.Source: Visual Understanding · p. 12

2.1 Overview

  • The overview groups representation learning into label supervision, language supervision, and image-only self-supervision, with CLIP and ALIGN using web-scale image-text pairs.Image-only methods include contrastive, non-contrastive, and masked image modeling approaches.Source: §2.1 · pp. 12–13

2.2 Supervised Pre-training

  • Supervised pre-training on ImageNet and ImageNet21K produces transferable representations and advances backbone architectures, but human annotation limits supervision scale and diversity.Web-derived noisy labels enable larger datasets such as JFT and I2E, while JFT-3B supports large-scale vision-transformer training and LiT enables zero-shot classification.Source: §2.2 · pp. 13–14

2.3.1 Basics of CLIP Training

  • CLIP and related models map images and text into a shared embedding space, achieving zero-shot image classification and image-text retrieval.Training scales batch size, data size, and model size, commonly using 16k–32k batches, billions of image-text pairs, and 300M–1B-parameter vision transformers.Source: §2.3.1 · pp. 15–16
  • CLIP reformulates zero-shot classification as retrieval using label semantics, and its aligned embedding space enables open-vocabulary detection and segmentation.The same design directly supports zero-shot image-text retrieval.Source: §2.3.1 · pp. 15–16

2.3.2 CLIP Variants

  • CLIP variants improve data, encoders, interpretability, modalities, and objectives, including masking, enriched or rewritten text, sparse embeddings, six-modality alignment, and fine-grained word-patch matching.Examples include FLIP, K-Lite, LaCLIP, STAIR, ImageBind, and FILIP.Source: §2.3.2 · pp. 16–17
  • Contrastive loss remains powerful at scale, while CoCa adds captioning for multimodal fusion, captioning learners show competitive scaling, and sigmoid loss achieves strong zero-shot classification.CoCa supports image captioning and VQA; Cap/CapPa argues captioners can be scalable vision learners; Zhai et al. (2023) avoids global similarity normalization with pairwise sigmoid loss.Source: §2.3.2 · pp. 17–18, Figure 2.6 · p. 17

2.4 Image-Only Self-Supervised Learning

Image-only self-supervised learning comprises contrastive learning, non-contrastive learning, and masked image modeling. These methods learn visual representations through agreement, architectural or statistical alternatives to negatives, or prediction of masked image content.

2.4.1 Contrastive and Non-contrastive Learning

  • Contrastive learning promotes positive pairs and repels negative pairs, with InfoNCE interpretable as maximizing a lower bound on mutual information between data views.Typical methods augment one image twice, use an encoder and projection head trained with contrastive loss, then discard the projection head for downstream tasks.Source: §2.4.1 · p. 18
  • Contrastive learning requires many negatives, supplied by memory banks, large batches, or MoCo’s queue and momentum encoder, while non-contrastive methods avoid negatives through asymmetric architectures, decorrelation, or clustering.SimSiam uses two augmented views, a prediction MLP, and stop-gradient without negative pairs or a momentum encoder; DINO instead uses student-teacher self-distillation with exponential-moving-average teacher updates.Source: §2.4.1 · pp. 18–19

2.4.2 Masked Image Modeling

  • Masked image modeling (MIM), introduced for image transformers by BEiT (Bao et al., 2022), masks image content and predicts targets such as discrete visual tokens, pixels, or features.The procedure uses an image tokenizer when predicting discrete tokens and can be viewed as knowledge distillation from a tokenizer teacher to a student that sees only part of the image.Source: §2.4.2 · pp. 19–20, Figure 2.8 · p. 19
  • MIM targets range from low-level pixels and HOG features to discrete tokens and high-level features from pretrained or momentum-updated teachers, with losses chosen according to target type.Cross-entropy is typically used for discrete tokens, whereas ℓ1, ℓ2, or cosine similarity losses are common for pixels and continuous-valued features.Source: §2.4.2 · pp. 20–21, Figure 2.9 · p. 20
  • MAE (He et al., 2022a) masks about 75% of image patches and is especially effective for object detection and segmentation, while video extensions can reach an optimal masking ratio of 90%.Video MIM extends MAE by masking space-time patches and reconstructing pixels; spacetime-agnostic random masking performed best in the cited study.Source: §2.4.2 · pp. 20–21
  • Vanilla MIM provides useful initialization but lacks global image representations, and although it scales to billion-parameter vision transformers, billion-scale image-data scaling remains unresolved.iBOT (Zhou et al., 2021) and DINOv2 (Oquab et al., 2023) address global representation learning with DINO-like self-distillation, while data-scaling studies remain mostly limited to millions of images.Source: §2.4.2 · p. 21

2.5 Synergy Among Different Learning Approaches · 2.6 Multimodal Fusion, Region-Level and Pixel-Level Pre-training

The paper surveys how CLIP is combined with label supervision, image-only learning, and masked image modeling, then reviews second-stage pre-training for multimodal fusion, region-level understanding, and pixel-level segmentation. These approaches extend global image-text representations toward deeper interaction, open-set detection, and promptable general-purpose segmentation.

2.5 Synergy Among Different Learning Approaches

  • CLIP can combine image-label and image-text supervision through unified contrastive learning, open-vocabulary adaptation, or joint classification and contrastive training.UniCL (Yang et al., 2022a) learns in a joint image-text-label space, LiT (Zhai et al., 2022b) adds a text tower to a supervised ViT-g/14 encoder, and MOFI (Wu et al., 2023d) uses entity labels and descriptions.Source: §2.5 · p. 22
  • Combining CLIP with image-only self-supervision improves transfer or data efficiency, but non-contrastive pre-training alone is insufficient for strong zero-shot classification.SLIP (Mu et al., 2021) combines SimCLR and CLIP, DeCLIP (Li et al., 2022g) mines modality-specific self-supervision, and xCLIP combines non-contrastive regularization with original CLIP training.Source: §2.5 · p. 22
  • CLIP features provide semantic targets for masked image modeling, while EVA-02 (Fang et al., 2023) alternates MIM and CLIP training to support both downstream fine-tuning and zero-shot classification.MVP (Wei et al., 2022b) directly regresses CLIP features, BEiTv2 (Peng et al., 2022a) distills them into discrete visual tokens, and EVA (Fang et al., 2023), CAEv2 (Zhang et al., 2022c), and MaskDistill (Peng et al., 2022b) use related targets.Source: Figure 2.11 · p. 22, §2.5 · p. 23
  • Joint CLIP-MIM training has not yet shown consistent gains at scale, whereas BEiT-3 (Wang et al., 2022d) performs masked data modeling across image, text, and joint image-text inputs with a multiway transformer.BEiT-3 achieves state-of-the-art performance across a wide range of vision and vision-language tasks.Source: §2.5 · p. 23, Figure 2.12 · p. 23

2.6.1 From Multimodal Fusion to Multimodal LLM

  • Because CLIP handles image-text interaction through a dot product, it supports zero-shot classification and retrieval but requires fusion encoders for captioning and visual question answering.Early OD-based methods use detector features and co-attention or soft prompts, while end-to-end methods increasingly use ViT-based encoders; representative models include UNITER (Chen et al., 2020d) and CoCa (Yu et al., 2022a).Source: §2.6.1 · pp. 23–24, Figure 2.13 · p. 24
  • Multimodal pre-training has shifted from masked-language, matching, and contrastive objectives toward PrefixLM and multimodal LLMs, including large-scale models and LLM-based instruction tuning.SimVLM (Wang et al., 2022g) introduced PrefixLM pre-training, followed by models such as Flamingo (Alayrac et al., 2022), PaLI (Chen et al., 2022h), LLaVA (Liu et al., 2023c), and MiniGPT-4.Source: §2.6.1 · p. 24

2.6.2 Region-Level Pre-training

  • Region-level pre-training extends CLIP toward fine-grained open-set detection by distilling CLIP knowledge, reformulating detection as phrase grounding, or unifying grounding with vision-language understanding.ViLD (Gu et al., 2021) and RegionCLIP use detector-based distillation; MDETR, GLIP, GLIPv2, and FIBER (Dou et al., 2022a) use grounded or unified pre-training.Source: §2.6.2 · p. 25, Figure 2.14 · p. 25

2.6.3 Pixel-Level Pre-training

  • SAM (Kirillov et al., 2023) builds a promptable segmentation foundation model with an MAE-pretrained image encoder, prompt encoder, lightweight mask decoder, and model-in-the-loop data engine.Its promptable task accepts points, boxes, masks, or free-form text, enabling adaptation to tasks including edge detection, object proposals, instance segmentation, and open-vocabulary segmentation; subsequent models broaden prompts and applications.Source: §2.6.3 · pp. 25–26, Figure 2.15 · p. 26

Visual Generation

This section surveys visual generation conditioned on human intents, focusing on text-to-image foundations and alignment methods. It covers spatial control, text-based editing, prompt adherence, and visual concept customization, and discusses unified alignment tuning across image-text generation tasks.

3.1.1 Human Alignments in Visual Generation

  • T2I alignment research targets four weaknesses: spatial controllability, text-based editing, faithful prompt following, and visual concept customization.Vanilla T2I models may overlook text descriptions, while intricate visual concepts are difficult to express verbally and may require specialized token embeddings or conditioned images.Source: §3.1.1 · pp. 27–29

3.1.2 Text-to-Image Generation

  • T2I generation seeks high-quality images that semantically correspond to input text, using GANs, VAEs, discrete token prediction, or diffusion models.Stable Diffusion exemplifies latent diffusion with a VAE, text encoder, denoising U-Net, and cross-attention for image-text interaction; its latent representation is 48 times smaller than the image.Source: §3.1.2 · pp. 29–31

3.2 Spatial Controllable Generation

  • Spatially controllable T2I extends text prompts with region descriptions, dense conditions, or inference-time guidance to specify image content and layout.ReCo uses position tokens and finetuning for region grounding; ControlNet adds a trainable branch for conditions such as edges, poses, masks, and depth; inference-time guidance avoids extra training but may be less precise than finetuning.Source: §3.2 · pp. 31–33
  • Spatial-control research is converging on integrating textual and spatial inputs, with region-controlled methods using position tokens and ControlNet-derived methods supporting dense image-like conditions.Follow-up models unify multiple conditions, while Disco separates background and human-pose controls to achieve high-fidelity foregrounds and backgrounds with arbitrary composition of subjects, backgrounds, and dance movements.Source: §3.2 · pp. 32–33

3.3 Text-based Editing

  • Text-based editing modifies selected image content or style while preserving most of the source image, using latent manipulation, cross-attention editing, or instruction-conditioned models.SDEdit and Blended Latent Diffusion manipulate diffusion latents; Prompt2Prompt edits word swaps, added phrases, and attention weights; Imagic edits natural images through blended text embeddings and test-time finetuning, while InstructPix2Pix accepts an image and language instruction.Source: §3.3 · pp. 33–34
  • Text-based editing has progressed from user-provided masks toward instruction-driven editing and future unified image-text generative models.InstructPix2Pix generates over 450K synthetic editing samples, while external segmentation, language, vision, and editing tools support more flexible workflows.Source: §3.3 · p. 35

3.4 Text Prompts Following

  • T2I models often omit objects, misapply attributes, or produce incorrect counts and relationships, motivating inference-time attention manipulation and alignment tuning.StructureDiffusion balances attention across parsed noun phrases, Attend-and-Excite emphasizes neglected subject tokens, and DDPO uses vision-language similarity rewards for finetuning.Source: §3.4 · pp. 36–37

3.5 Concept Customization

  • Visual concept customization evolves from test-time token or model finetuning to in-context generation conditioned on subject images without test-time finetuning.SuTI generalizes to unseen subjects and descriptions, while InstantBooth generates language-aligned images that preserve identities in a single forward pass.Source: §3.5 · pp. 38–39

3.6 Trends: Unified Tuning for Human Alignments

  • The envisioned unified alignment-tuned T2I model would process content text, image conditions, and instructions to support spatial control, editing, customization, and mixed tasks.Its second-stage training would use alignment-focused data, losses, and rewards, potentially balancing aesthetics, text alignment, safety, and instruction adherence.Source: §3.6 · pp. 40–41
  • Future work may close the loop between multimodal understanding and generation through image-text-input, image-text-output foundation models.The appropriate balance and efficient integration of understanding and generation remain open questions.Source: §3.6 · p. 41

Unified Vision Models

This chapter surveys the unification of vision models, covering its challenges, recent efforts, and future trends. It focuses on open-set models, unified task granularities, and more promptable vision interfaces.

Unified Vision Models

  • The chapter introduces challenges and recent efforts toward unifying vision models before discussing future trends.These topics are covered in Sections 4.1 and 4.5.Source: Unified Vision Models · p. 42
  • It examines how to transform closed-set vision models into open-set ones.This topic is addressed in Section 4.2.Source: Unified Vision Models · p. 42
  • It studies unifying different vision-task granularities and building a more promptable interface for vision.These topics are covered in Sections 4.3 and 4.4.Source: Unified Vision Models · p. 42

4.1 Overview

The pursuit of general-purpose vision systems is motivated by NLP’s shift from task-specific models to unified, instruction-following interfaces, but vision faces greater diversity in inputs, granularities, outputs, and data costs. Current efforts therefore focus on bridging vision and language, unified multi-task modeling, and LLM-like promptable interfaces.

Overview

  • The proposed vision-system agenda is inspired by NLP’s progression from task-specific models to Transformer-based unified language models and instruction-following systems such as ChatGPT 1.This evolution motivates investigating whether vision can achieve a comparable general-purpose system.Source: §4.1 · p. 42

Challenges

  • Vision unification is difficult because tasks span images, videos, multimodal inputs, image-, region-, and pixel-level granularities, heterogeneous outputs, and uneven annotation and data costs.Vision data is generally more expensive to collect than text data, making vision corpora often smaller than text corpora.Source: §4.1 · p. 42

Towards a unified vision model

  • Recent unified vision efforts pursue three directions: open-set vision-language models, unified multi-task modeling, and promptable interfaces.These directions correspond to making vision models open-set, unifying task granularities, and enabling interaction through prompts.Source: §4.1 · p. 42, Figure 4.1 · p. 43

Bridging vision and language

  • Vision-language models such as CLIP extend closed-set classification to open-world recognition by mapping visual signals to rich semantics, enabling zero-shot transfer and open-vocabulary recognition.The cited applications include diverse vision recognition tasks.Source: §4.1 · p. 43

Unified multi-task modeling

  • Unified multi-task modeling aims to replace costly task-specific development with a single vision model that performs well across many vision tasks.Traditional models require task-specific data, making new-task development prohibitively expensive.Source: §4.1 · p. 43

LLM-like promptable interface

  • An LLM-like promptable vision interface should use language or in-context prompts to produce user-intended outputs without changing model parameters.This capability would align outputs with varied user intents without finetuning.Source: §4.1 · p. 43

4.2 From Closed-Set to Open-Set Models

Open-set vision models replace fixed label vocabularies with language-aligned visual-semantic spaces, enabling zero-shot transfer across recognition tasks. The section organizes these models by initialization, architecture, pre-training, and applications to object detection, grounding, segmentation, and referring segmentation.

4.2 From Closed-Set to Open-Set Models

  • CLIP converts closed-set recognition into open-set recognition by aligning images with textual descriptions in a shared semantic space.Because textual concepts share a coherently pretrained feature space without a predefined vocabulary, the representation transfers to diverse image-recognition tasks in a zero-shot manner.Source: §4.2 · pp. 43–44, Figure 4.2 · p. 44
  • Open-set models vary in initialization, design, and pre-training, using CLIP initialization or augmentation, multi-stage or end-to-end architectures, and supervised, semi-supervised, or weakly supervised learning.These approaches extend visual-language understanding from image-level recognition to region-level localization and pixel-level grouping, while separately pretrained or jointly learned representations support efficient training.Source: §4.2 · pp. 44–45

4.2.1 Object Detection and Grounding

  • Open-set object detection evaluates transfer beyond training vocabularies through zero-shot, strict open-vocabulary, and generalized open-vocabulary settings.Object grounding generalizes open-set detection by localizing image objects associated with noun phrases, while GLIP, DetCLIPv2 (Yao et al., 2023), and Grounding-DINO (Liu et al., 2023h) learn language-conditioned localization from multimodal supervision.Source: §4.2.1 · pp. 46–47, Figure 4.3 · p. 46

4.2.2 Image Segmentation and Referring

  • Open-vocabulary segmentation transfers foundation-model visual-semantic knowledge to pixel-level tasks, while referring segmentation directly localizes masks from language queries.FC-CLIP (Yu et al., 2023a) uses a convolutional CLIP backbone, ODISE (Xu et al., 2023a) exploits diffusion-model features, and X-Decoder unifies segmentation tasks with a generalized encoder-decoder.Source: §4.2.2 · pp. 47–48
  • A central challenge for open-vocabulary segmentation is limited semantically annotated data, prompting methods to add detection supervision and datasets with broader, denser, or higher-quality masks.Most existing methods still rely on COCO segmentation annotations, while OpenSeeD (Zhang et al., 2023e), DataSeg (Gu et al., 2023), and new multi-domain or large-scale datasets address supervision gaps.Source: §4.2.2 · p. 48, Figure 4.5 · p. 48, Figure 4.6 · p. 49

4.3 From Task-Specific Models to Generic Models

Vision tasks differ across space, time, and modality, while annotation scales vary substantially, limiting task-specific models’ ability to exploit cross-task synergy. Recent generic models address this challenge through I/O unification or functionality unification, with I/O unification inspired by LLM-style sequential modeling.

4.3 From Task-Specific Models to Generic Models

  • Vision tasks span image-, region-, and pixel-level granularity, static images and video, and heterogeneous input-output modalities, impeding unified modeling.These modalities include images, text, human pose, and depth maps.Source: §4.3 · p. 49
  • Annotation scales differ sharply across tasks, from up to 2B image-text samples in LAION to 1.7M detection images in Objects365, with segmentation especially costly.The data-scale pyramid reflects differing annotation semantics and the high cost of mask annotation.Source: §4.3 · p. 49
  • Generic vision models group into I/O unification, which reformulates tasks as sequence-to-sequence prediction, and functionality unification, which uses coherent encoder-decoder architectures.Both approaches aim to exploit synergy across tasks with different granularities.Source: §4.3 · pp. 49–50

4.3.1 I/O Unification

4.3 From Task-Specific Models to Generic Models

  • Functionality unification predicts heterogeneous semantic or spatial outputs and combines them through routers or heads, avoiding task- or modality-specific tokenizers but requiring sophisticated model design (Yu et al., 2022a).The two unification strategies differ in whether decoding begins from homogeneous token sequences or heterogeneous functional outputs.Source: §4.3 · pp. 49–50, Figure 4.7 · p. 50

Sparse and discrete outputs

Sparse and discrete vision outputs can be unified with language modeling by tokenizing text, numeric coordinates, and visual structures into autoregressively decoded sequences. Representative systems extend this interface from multimodal task unification to LLM-augmented reasoning and localization.

Sparse and discrete outputs

Dense and continuous outputs

Dense and continuous vision outputs are difficult to discretize, motivating image-oriented tokenizers and unified decoding models. Functionality unification instead groups tasks by shared output characteristics, including location, semantic, and pixel-level outputs.

Dense and continuous outputs

  • Dense outputs such as segmentation, depth estimation, inpainting, and editing generally require image-oriented tokenizers because their high-dimensional values resist discrete-token conversion.Segmentation masks are an exception because they can be approximated by polygons; VQ-VAE encodes images into discrete codebook tokens for decoding.Source: Dense and continuous outputs · p. 52
  • VQ-GAN extends VQ-VAE with patch-level adversarial regularization and perceptual loss to balance high-resolution output quality with compression efficiency.The discriminator regularizes image decoding at the patch level.Source: Dense and continuous outputs · p. 52
  • UViM unifies panoptic segmentation, depth estimation, and colorization through VQ-VAE-based dense decoding, then learns a language model to generate oracle codes from input images.Its two-stage process first trains an encoder-decoder and restricted oracle, then replaces desired-output oracle inputs with predicted codes.Source: Dense and continuous outputs · p. 52
  • Unified-IO pretrains task-specific VQ-VAEs, jointly trains an encoder-decoder on 90 datasets, reaches state of the art on GRIT, and remains weaker than leading models on common tasks.Follow-up methods use soft tokens and masked modeling, while inpainting-based approaches and Painter extend task unification with competitive benchmark performance.Source: Dense and continuous outputs · pp. 52–53, Figure 4.11 · p. 53

4.3.2 Functionality Unification

  • Functionality unification organizes vision tasks by shared characteristics rather than treating them as fully isolated or aligned, distinguishing location, semantic, and pixel-level outputs.Detection and phrase grounding share localization outputs, segmentation tasks share masks, and other tasks produce concepts or textual descriptions.Source: §4.3.2 · p. 53

Multi-task learning

Multi-task learning has progressed from task-specific vision architectures toward unified multimodal models, but early approaches generally combine tasks without exploiting strong cross-task synergy. Recent work instead seeks functional unification that better leverages relationships among vision and vision-language tasks.

Multi-task learning

  • Early multi-task learning unified different vision or vision-language tasks, establishing the foundation for shared-task modeling.Source: Multi-task learning · p. 53
  • UberNet applies one budget-controllable CNN to seven vision tasks, while Taskonomy (Zamir et al., 2018) models task affinities through latent-space transfer.Taskonomy finds distinct relationships across task groups, including surface normal estimation with reshaping and point matching, and curvature extraction with image segmentation.Source: Figure 4.12 · p. 54, Multi-task learning · p. 54
  • 12in1 (Lu et al., 2020) combines 12 vision-language tasks in one BERT-based model, improving over single-task learning while substantially reducing parameters.It uses task-specific heads with a shared ViLBERT trunk.Source: Multi-task learning · p. 54
  • UniT (Hu and Singh, 2021b), E2E-VLP (Xu et al., 2021), and GPV (Gupta et al., 2022b) extend end-to-end encoder-decoder modeling across vision and vision-language tasks.These systems use task-specific or modality-specific decoding components, with UniT avoiding reliance on pre-trained detectors and GPV combining DETR features, cross-attention, and vision/language decoders.Source: Multi-task learning · p. 54
  • Because modular heads and task-specific components rarely create cross-task synergy, newer models such as X-Decoder (Zou et al., 2023a) pursue functional unification with one shared decoder and task routing.Source: Multi-task learning · p. 54, Figure 4.13 · p. 55

Unified learning

Unified learning uses a single model to bind vision and vision-language inputs in a shared semantic space, enabling zero-shot task support and task-specific fine-tuning. Representative systems achieve this through unified pretraining and shared encoder-decoder interfaces across tasks.

Unified learning

  • Transformers and open-set models enable unified vision systems to support many vision and vision-language tasks zero-shot, beyond serving solely as representation-learning engines.Task-specific fine-tuning can further improve performance after pre-training.Source: Unified learning · p. 55

GLIPv2

  • GLIPv2 unifies localization and vision-language pretraining through phrase grounding, region-word contrastive learning, and masked language modeling.Its fused localization and vision-language matching modules support plausible zero-shot performance across tasks and improve training efficiency with fused data.Source: Unified learning · p. 55

X-Decoder

  • X-Decoder unifies tasks with latent and text queries, shared pixel-mask and semantic-embedding outputs, a common text encoder, and decoupled image-text encoders.Different task routes share the same parameters while learning from intra-image and inter-image supervision across task granularities.Source: Unified learning · p. 55

Uni-Perceiver-v2

  • Uni-Perceiver-v2 combines vision and text encoders with a general decoder, encoding predicted boxes and masks as queries.A unified maximum-likelihood strategy jointly trains localization and non-localization tasks across datasets with different levels of supervision.Source: Unified learning · p. 56

4.4 From Static to Promptable Models

Vision models are becoming more promptable by supporting multimodal inputs and in-context interaction, addressing their limited flexibility relative to language models. Recent approaches use spatial, visual, and cross-modal prompts, while in-context methods remain unable to jointly handle multimodal inputs and diverse output types in one model.

4.4 From Static to Promptable Models

  • Promptable vision models extend static systems with multimodal prompting and in-context prompting to support more flexible human interaction.Unlike language models, vision systems require prompts beyond language to resolve spatial and visual ambiguity.Source: §4.4 · p. 56, §4.4.1 · p. 56

4.4.1 Multi-modal Prompting

  • SAM (Kirillov et al., 2023) accepts points or boxes and segments images at arbitrary granularity, establishing a flexible spatial prompting interface.Its foundation model and annotation engine support downstream promptable segmentation applications, including medical imaging.Source: §4.4.1 · p. 56, Figure 4.14 · p. 57
  • SEEM (Zou et al., 2023b) combines points, boxes, strokes, and reference images through a visual sampler for zero-shot image and video object segmentation.The visual sampler extracts features at user-specified locations without requiring additional training for reference-image prompting.Source: §4.4.1 · p. 57, Figure 4.15 · p. 57
  • Other multimodal prompting methods unify diverse inputs, including Painter’s image-and-annotation prompts, Prismer’s extracted vision features, and ImageBind’s cross-modal embedding alignment.ImageBind aligns image/video, language, audio, and depth so different prompt types can be composed by summing embeddings.Source: §4.4.1 · p. 57

4.4.2 In-context Prompting

  • No single model yet accepts multimodal inputs and predicts diverse output types in an in-context manner across language and vision tasks.Flamingo (Alayrac et al., 2022) and Kosmos-1 (Huang et al., 2023b) support multimodal inputs but primarily generate text, whereas vision-only methods target tasks such as segmentation and depth estimation.Source: §4.4.2 · pp. 58–59
  • Visual in-context learning reformulates dense prediction as prompted image completion, with Bar et al. (2022) predicting edges, masks, and depths from masked images.The method uses discrete VQ-GAN tokens, a ViT encoder, and an 88k-image Computer Vision Figures dataset.Source: §4.4.2 · p. 58, Figure 4.16 · p. 58
  • Painter (Wang et al., 2023i) unifies task inputs and outputs in pixel space, while SegGPT (Wang et al., 2023j) extends this framework to exemplar-based image segmentation.Painter uses task-specific image-space encodings with vanilla ViT and masked image modeling; SegGPT specializes the approach for segmentation.Source: §4.4.2 · p. 58, Figure 4.17 · p. 59
  • Hummingbird (Balažević et al., 2023) performs dense in-context prediction by retrieving nearest-neighbor features from labeled prompt images and transferring their labels.Its mechanism applies K-nearest-neighbor-style matching to dense prediction tasks.Source: §4.4.2 · pp. 58–59, Figure 4.18 · p. 59

4.5 Summary and Discussion

The chapter surveys a trend toward open-world, unified, interactive vision models while highlighting fundamental differences between vision and language that create unresolved scaling and integration questions. It concludes that intelligent human-like vision systems remain distant, despite substantial progress and effort.

Summary

  • Vision requires specialized tokenization, annotation, and heterogeneous-data handling because it captures raw physical signals unlike language.Vision data is also substantially more expensive to store: ImageNet exceeds hundreds of gigabytes, while Howto100M surpasses GPT-3’s 45 TB training corpus.Source: §4.5 · pp. 59–60
  • The field is moving toward open-world, unified, and interactive vision models, as summarized in Figure 4.19.The chapter identifies unification across different aspects as a route toward more intelligent vision systems.Source: §4.5 · p. 59, Figure 4.19 · p. 60

Open Questions

  • Open questions concern computer vision in the wild, scaling laws for vision, and whether future systems should be vision-centric or language-centric.Current models struggle with novel and long-tail scenarios, the correct scaling path and emerging properties remain unclear, and the appropriate balance between vision models and LLMs is unresolved.Source: §4.5 · p. 60

Conclusion

  • The review presents existing efforts as progress toward, but not yet achievement of, vision systems that perceive the world like humans.Its purpose is to provide an overall picture of the field and inspire next-generation vision models.Source: §4.5 · p. 60

Large Multimodal Models: Training with LLM

This section traces the development of large multimodal models from image-to-text generation and multimodal in-context learning to instruction-tuned systems, surveying multimodal extensions, emerging modalities, and parameter-efficient adaptation. It also examines instruction data, multitask supervision, benchmarks, and applications while identifying instruction-following, multimodal alignment, and scaling as ongoing challenges.

5.1.1 Image-to-Text Generative Models

  • Image-to-text LMMs generally combine an image encoder, an optional trainable connector, and an autoregressive language model that predicts text tokens conditioned on image and prior-text tokens.GIT trains a contrastively pretrained image encoder with a language model from scratch, whereas BLIP2 freezes both pretrained models and trains a lightweight Q-former.Source: §5.1.1 · p. 61, §5.1.2 · p. 62

5.1.2 Case Studies

  • Flamingo demonstrates multimodal in-context learning, transferring to unseen tasks such as visual math from a few image-text examples without additional training or task-specific tuning.It connects frozen pretrained vision and language models with Perceiver Sampler and Gated Transformer modules and trains on web-scale interleaved multimodal data.Source: §5.1.2 · p. 62

5.1.3 OpenAI Multimodal GPT-4 and Research Gaps

  • GPT-4 (OpenAI, 2023a) adds visual understanding and reasoning to earlier LLM properties, exposing a research gap in multimodal instruction-following and alignment.Earlier GPT models established in-context learning and chain-of-thought reasoning, while ChatGPT and InstructGPT emphasized instruction-following and alignment with human intents.Source: §5.1.3 · p. 64

5.2 Pre-requisite: Instruction Tuning in Large Language Models

  • Instruction tuning converts implicit task data into instruction-input-output triplets, enabling one model to compose learned tasks and generalize more naturally to unseen instructions.Traditional task-specific or multitask training lacks explicit task instructions and therefore makes zero-shot task distinction and generalization difficult.Source: §5.2 · p. 65

5.2.1 Instruction Tuning

  • Self-instruct uses a few human-created seeds and an LLM such as GPT-3 or GPT-4 to generate additional instruction-following data at lower cost than human annotation.The approach improves zero-shot and few-shot generalization, and supports open models including Alpaca and Vicuna; however, imitation models can retain a substantial capability gap from closed LLMs.Source: §5.2.1 · pp. 65–66, §5.2.2 · p. 67

5.3 Instruction-Tuned Large Multimodal Models

  • LLaVA uses GPT-4-generated multimodal instructions, connects CLIP ViT-L/14 to Vicuna through a projection layer, and trains in two stages from feature alignment to end-to-end fine-tuning.The instruction data contains multi-turn conversation, detailed description, and complex reasoning examples, totaling 158K samples.Source: §5.3 · p. 68, Data Creation · p. 68, Network Architecture and Training · p. 68

Performance

  • 92.53% accuracy is achieved on Science QA when GPT-4 judges and combines LLaVA’s answers, exceeding LLaVA alone at 90.92%.LLaVA also reaches 85.1% relative score on LLaVA-Bench (COCO) and 73.5% on LLaVA-Bench (In-the-Wild), compared with 77.8% for Google Bard and 71.5% for BingChat on the latter.Source: Performance · pp. 69–70

More Modalities (Beyond VL)

  • LMM research is expanding from vision-language systems toward models that incorporate speech, audio, video, 3D, sensors, and arbitrary combinations of multimodal inputs and outputs.ImageBind supplies a shared embedding space for several modalities, while NExT-GPT combines an LLM with multimodal adaptors and diffusion decoders.Source: More Modalities (Beyond VL) · p. 70, §5.4 · p. 70

Improving Visual Instruction Data Quantity and Quality

  • Instruction-tuning quality can matter more than quantity: InstructionGPT-4 selects 200 high-quality samples from MiniGPT-4’s 3,500 and fine-tunes a better model.Other approaches expand data with region descriptions, negative instructions, OCR examples, synthesized images, or model-generated and rewritten instructions; SVIT scales to 3.2 million examples.Source: Improving Visual Instruction Data… · pp. 72–73

Multitask Instruct with Established Academic Datasets/Tasks

  • InstructBLIP converts 26 public datasets into instruction format and achieves state-of-the-art zero-shot performance on all 13 held-out datasets, outperforming BLIP-2 and larger Flamingo models.Qwen-VL also scales image-text pair and academic multitask pre-training to achieve strong performance across many tasks.Source: Multitask Instruct with Established… · p. 73

Multimodal In-Context-Learning

  • Multimodal in-context learning steers models with a few image-response examples, and Otter strengthens OpenFlamingo using 2.4 million MIMIC-IT instances with in-context examples.With two image-text pairs as context, Otter learns the concise response style demonstrated by the examples.Source: Multimodal In-Context-Learning · pp. 73–74

Parameter-Efficient Training

  • Parameter-efficient training freezes most parameters and updates only a small fraction, making LMM adaptation practical under limited GPU resources.LLaMA Adapter v2 and LAVIN train 14M and 3.8M parameters against 7B/13B models; QLoRA fine-tunes 65B LLaMA for 24 hours on one GPU at 99.3% of ChatGPT’s performance level, while LoRA/QLoRA can match full-model tuning for scaled LLaVA.Source: Parameter-Efficient Training · p. 74

Benchmarks

  • LMM evaluation spans OCR, hallucination, adversarial robustness, and comprehensive benchmarks, with LLaVA-Bench introducing open-world visual-chat evaluation.Across 24 text-recognition datasets, LLaVA outperforms MiniGPT-4 on 21 despite using an order of magnitude less training data; more training data improves OCR, although supervised state-of-the-art usually remains stronger, with BLIP2 surpassing it on WordArt.Source: Benchmarks · p. 74

Applications

  • Domain-specific assistants can improve helpfulness through specialized data, reduce model and serving costs, and keep sensitive prompts local; LLaVA-Med adapts LLaVA to biomedical images with curriculum learning and GPT-4-generated instructions.LLaVA-Med answers biomedical questions precisely, whereas LLaVA often responds like a layperson and hallucinates from commonsense; related biomedical models include PaLM-M, Stanford Med-Flamingo, and radiology generalist.Source: Applications · pp. 74–76

5.5 How Close We Are To OpenAI Multimodal GPT-4?

  • Open-source systems are close to multimodal GPT-4 in enabling new prototype capabilities, including multimodal chat and end-to-end image generation, but remain far behind in scaling visual reasoning.GPT-4-level examples require understanding multiple high-resolution images, long text sequences, and domain knowledge, demanding compute and language models unavailable to most users; future work should combine capability scaling with prototypes, evaluation, and reduced computational barriers.Source: §5.5 · p. 76

Multimodal Agents: Chaining Tools with LLM

This section surveys multimodal agents as a training-free paradigm that chains LLMs with external multimodal tools to support complex reasoning, actions, and diverse applications. It covers their evolution, frameworks, capabilities, extensibility, tradeoffs, tool-use accuracy, evaluation, tool creation, retrieval-augmented capabilities, tool allocation, and external-knowledge access.

6.1 Overview

  • Multimodal agents shift modeling from task-specific and pretrained-then-finetuned systems toward general-purpose assistants that use LLMs to chain existing tools, often without additional training.This paradigm addresses LLM limitations in mathematical reasoning, information retrieval, outdated knowledge, and multimodal perception by supplementing them with external models, APIs, and code interpreters.Source: §6.1 · pp. 77–78

6.2 Multimodal Agent

  • A typical multimodal agent uses an LLM as a tool allocator that plans and decomposes requests, selects and executes tools over multiple rounds, then feeds serialized observations back for response generation.Its core components are tools for missing information, planning through few-shot prompting or fine-tuning, and execution through regex matching, generated programs, or natural-language tool instructions.Source: §6.2 · pp. 79–80

6.3 Case Study: MM-REACT

  • MM-REACT (Yang* et al., 2023) composes ChatGPT with vision tools for multimodal reasoning and action across text, images, and video.The system uses tools such as captioning, OCR, celebrity recognition, object detection, dense captioning, tagging, face detection, and Bing Search, with ChatGPT planning calls and integrating observations.Source: §6.3 · p. 80, §6.3.1 · pp. 81–82

6.3.2 Capabilities

  • MM-REACT demonstrates visual math and text reasoning, joke and meme understanding, spatial reasoning, visual planning, multi-image reasoning, document understanding, open-world concepts, and video analysis.These capabilities include multi-hop reasoning and summarization over multimodal inputs, extending beyond the scenarios readily achievable through visual instruction tuning alone.Source: §6.3.2 · p. 83

6.3.3 Extensibility

  • Tool-chaining systems can upgrade their LLMs and add tools without retraining, with MM-REACT potentially improving by replacing ChatGPT with GPT-4 and other systems scaling toward hundreds or millions of tools.HuggingGPT, Chameleon, RestGPT, and TaskMatrix.AI exemplify progressively broader tool integration, while InternGPT adds click, scribble, and bounding-box interactions through SAM.Source: §6.3.3 · pp. 84–85

6.4 Advanced Topics

  • Compared with instruction tuning, tool chaining avoids data curation and training costs but may select inappropriate tools or propagate noisy expert outputs, motivating code generation, self-assessment, and instruction tuning for more accurate planning.Code-based methods still inherit LLM execution errors; AssistGPT adds an inspection-and-learning loop, while other methods tune smaller LLMs on instruction-API pairs.Source: §6.4.1 · p. 85, §6.4.2 · pp. 85–86

6.4.2 Improving Multimodal Agents

  • LMMs could replace LLMs as tool allocators, coordinating multimodal tools directly without converting multimodal outputs into text sequences.This could enable natural interactions, such as using multimodal GPT-4 (OpenAI, 2023a) to coordinate image and video generation tools for short-film creation.Source: §6.4.2 · p. 86

6.4.3 Diverse Applications of Multimodal Agents

6.4.4 Evaluation of Multimodal Agents

  • Evaluation remains underdeveloped: API-Bank (Li et al., 2023k) begins systematic tool-use assessment, while MM-Vet evaluates integrated capabilities with six core VL abilities and 16 capability combinations.MM-Vet (Yu et al., 2023d) also uses an LLM-based evaluator for open-ended outputs, addressing limitations of existing specialized benchmarks and conventional metrics.Source: Figure 6.9 · p. 86, §6.4.4 · p. 87, Figure 6.10 · p. 87

6.4.5 Tool Creation

  • A key open problem is creating multimodal tools on demand; proposed directions include LLM-generated programs using OpenCV and automated model-training pipelines such as AutoML GPT (Zhang et al., 2023j).CREATOR (Qian et al., 2023) and Cai et al. (2023) demonstrate tool creation and collaborative tool-making for non-multimodal tasks, motivating multimodal extensions.Source: §6.4.5 · p. 88

6.4.6 Retrieval-Augmented Multimodal Agents

  • Retrieval-augmented multimodal agents address long-tail and rapidly changing knowledge by retrieving external visual or textual information, including multimodal documents, for core-model use.RA-CM3 (Yasunaga et al., 2022) combines a multimodal retriever with a generator capable of producing text and images; images can also serve as queries to knowledge bases or vision experts.Source: §6.4.6 · p. 88

Conclusions and Research Trends

The paper surveys multimodal foundation models across specific-purpose models and general-purpose assistants, tracing their transition toward systems that follow human intents across diverse vision and vision-language tasks. It highlights research trends toward multimodal generalist agents with visual interaction, planning, memory, and tool use.

7.1 Summary and Conclusions

  • The survey organizes multimodal foundation models into specific-purpose models and general-purpose assistants.Specific-purpose models cover visual understanding and text-to-image generation, while assistants cover unified vision models, end-to-end training with LLMs, and chaining vision tools with LLMs.Source: §7.1 · p. 89
  • Specific-purpose models pre-train on large problem-related datasets and transfer to related real-world scenarios, supporting visual understanding and controllable text-to-image generation.Visual understanding operates at image, region, and pixel levels, while visual generation enables increasingly fine-grained user control and customization.Source: §7.1 · p. 89
  • General-purpose assistants pursue unified architectures, input-output formats, and human interfaces, using LLMs through unified vision modeling, end-to-end training, or tool chaining.The surveyed approaches borrow LLM unification, connect visual data to a pre-trained LLM, or freeze the LLM and prompt it to trigger vision experts.Source: §7.1 · p. 89

7.2 Towards Building General-Purpose AI Agents

  • The field is transitioning from specialist models toward general-purpose visual assistants, while existing systems such as Flamingo (Alayrac et al., 2022) and multimodal GPT-4 (OpenAI, 2023b) remain preliminary.The paper frames this transition as progress toward general-purpose multimodal AI agents built from foundation models.Source: §7.2 · p. 90
  • Future agents should fuse language, vision, speech, and actions, represent human intents through visual prompts, and support precise interaction for visual understanding and generation.Gato (Reed et al., 2022) and PaLM-E (Driess et al., 2023) illustrate single-weight systems spanning language, multimodal, and control tasks; SAM (Kirillov et al., 2023) and ControlNet/GLIGEN (Zhang and Agrawala, 2023; Li et al., 2023n) illustrate visual intent representations.Source: §7.2 · p. 90
  • Agent-oriented multimodal models require planning, memory, and tool use, including finer visual understanding, multimodal retrieval, and API selection for detection, segmentation, OCR, or generation.Planning involves decomposing tasks and self-improvement; memory spans in-context learning and long-term multimodal retrieval; tool use supplements model knowledge with external APIs.Source: §7.2 · pp. 90–91

Related papers

Loading 2309.10020v1…