PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Foundation Models: From Specialists to General-Purpose Assistants

Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao

arXiv:2309.10020v1cs.CVcs.CL

TL;DR

The paper surveys how multimodal foundation models support visual perception, generation, and interaction, while examining the gap between existing multimodal models and general-purpose assistants. It organizes established specialist models and emerging assistant-oriented approaches, highlighting multimodal in-context learning and open directions for integrated understanding and generation.

  • Problem

    The field seeks AI systems that can perceive, generate, reason over, and interact with visual signals, while existing multimodal models still lack instruction-following and alignment capabilities associated with multimodal GPT-4.

  • Method

    The paper provides a comprehensive survey organized around five topics spanning visual understanding, text-to-image generation, unified vision models, end-to-end multimodal LLMs, and multimodal tool use.

  • Results

    The survey documents a transition from problem-specific pretraining and transfer toward general-purpose multimodal assistants, including multimodal in-context learning that enables zero-shot transfer to unseen tasks.

  • Takeaways & Limitations

    Multimodal foundation models increasingly combine visual understanding with language-based task handling, but integrated image-text understanding and generation remain open research directions.

  • Takeaways & Limitations

    A fully integrated image-text-input, image-text-output foundation model and an efficient way to balance understanding with generation remain future explorations.

Abstract

from arXiv · show

This paper presents a comprehensive survey of the taxonomy and evolution of multimodal foundation models that demonstrate vision and vision-language capabilities, focusing on the transition from specialist models to general-purpose assistants. The research landscape encompasses five core topics, categorized into two classes. (i) We start with a survey of well-established research areas: multimodal foundation models pre-trained for specific purposes, including two topics -- methods of learning vision backbones for visual understanding and text-to-image generation. (ii) Then, we present recent advances in exploratory, open research areas: multimodal foundation models that aim to play the role of general-purpose assistants, including three topics -- unified vision models inspired by large language models (LLMs), end-to-end training of multimodal LLMs, and chaining multimodal tools with LLMs. The target audiences of the paper are researchers, graduate students, and professionals in computer vision and vision-language multimodal communities who are eager to learn the basics and recent advances in multimodal foundation models.

Introduction

The paper surveys multimodal foundation models for vision and vision-language, tracing a transition from task-specific pre-trained models toward unified models and general-purpose assistants. It organizes this landscape around visual understanding, visual generation, and emerging assistant-oriented approaches.

  • Motivation: Vision-capable AI aims to perceive, generate, reason over, and interact with visual signals, including recognizing scenes and creating images.Building foundational models with visual capabilities is presented as a central research direction.
  • Evolution: The field is moving from task-specific models trained for individual datasets and tasks toward broadly pretrained models that transfer across downstream tasks.The trajectory is described across language, vision, and multimodality, with large-scale pre-training enabling broader adaptation.
  • Specific-purpose models: Established multimodal foundation-model topics include visual representation learning and text-to-image generation.Visual understanding methods cover backbone learning and multimodal, region-level, and pixel-level pre-training; generation methods include controllable image synthesis.
  • General-purpose assistants: Emerging assistant-oriented topics include unified vision models, training multimodal models with LLMs, and chaining multimodal tools with LLMs.These directions target general-purpose assistants that follow human intents across varied computer-vision tasks.
  • Scope: Multimodal foundation models extend foundation-model ideas to systems that process vision and vision-language modalities.The paper categorizes these models by functionality and generality.
  • Learning signals: Language supervision from large-scale image-text pairs enables zero-shot image classification and open-vocabulary computer vision tasks.CLIP and ALIGN are cited as representative models trained contrastively on millions or billions of noisy web-mined image-text pairs.

Visual Understanding

Visual understanding methods learn general image representations through label supervision, language supervision, or image-only self-supervision, then extend pretrained encoders to multimodal, region-level, and pixel-level tasks.

  • Scope: General-purpose image representations support image-level, region-level, and pixel-level computer vision tasks.The surveyed downstream tasks include classification, retrieval, captioning, detection, grounding, and segmentation.
  • Learning paradigms: The chapter organizes image representation learning by label supervision, language supervision, and image-only self-supervision.Image-only methods include contrastive learning, non-contrastive learning, and masked image modeling.
  • Fine-grained understanding: The chapter also covers multimodal fusion, region-level grounding, and pixel-level segmentation methods that typically build on pretrained image or image-text encoders.Representative examples include CoCa, Flamingo, GLIP, and SAM.
  • Supervised pre-training: Supervised pre-training on datasets such as ImageNet and ImageNet21K has powered transferable backbones and many vision tasks, but annotation expense limits supervision scale and diversity.The approach maps images to discrete labels associated with visual concepts.
  • Supervised pre-training: Large-scale classification datasets can derive noisy labels from web image-text pairs, enabling models such as BiT and 22B-parameter vision transformers.JFT-300M trained BiT models, while JFT-3B supported scaling a plain vision transformer to 22B parameters.
  • Language supervision: CLIP uses language supervision and contrastive learning, with large batch sizes, billions of image-text pairs, and scaled model size.Its aligned image-text space supports zero-shot classification, image-text retrieval, and open-vocabulary detection and segmentation.

2.4 Image-Only Self-Supervised Learning

Image-only self-supervised learning comprises contrastive, non-contrastive, and masked image modeling approaches, differing in their use of negatives, architectural constraints, and reconstruction targets.

  • Overview: Image-only self-supervised learning is divided into contrastive learning, non-contrastive learning, and masked image modeling.These approaches derive supervision from the image itself rather than external labels or text.
  • Contrastive learning: Contrastive methods augment one image twice and train an encoder-projector to maximize agreement between corresponding views.The projection head is discarded for downstream tasks.
  • Contrastive learning: Contrastive learning requires many negative samples, maintained through memory banks, large batches, or queues with momentum encoders.This negative-sample requirement is a central caveat of the approach.
  • Non-contrastive learning: Non-contrastive methods avoid negative samples through asymmetric architectures, dimension decorrelation, or clustering.SimSiam uses two augmented views, a prediction MLP, and stop-gradient without negatives or a momentum encoder.
  • Masked image modeling: Masked image modeling corrupts image patches and predicts masked targets using pixels, discrete tokens, or continuous features.Image tokenizers convert images into discrete visual tokens for methods such as BEiT, while MAE reconstructs raw pixels.
  • Masked image modeling: MIM scales to billion-parameter vision transformers, but its data-scaling behavior at billion-image scale remains insufficiently explored.Existing studies generally use millions of images, with limited exceptions examining billion-scale data.

2.5 Synergy Among Different Learning Approaches

The surveyed methods combine label, language, image-only, and masked-image objectives, while deeper multimodal fusion and end-to-end multimodal models address capabilities that dual encoders lack.

  • Combining supervision: UniCL jointly uses image-label and image-text data in a unified image-text-label space for contrastive learning.Florence is described as a scaled-up version of UniCL.
  • Combining supervision: SLIP combines SimCLR with CLIP and outperforms CLIP in zero-shot transfer and linear-probe settings.DeCLIP adds self-supervised signals to improve CLIP training data efficiency.
  • Combining supervision: MVP and related methods use CLIP image features as MIM targets, while BEiTv2 compresses them into discrete visual tokens before BEiT training.EVA-02 alternates learning MIM and CLIP representations.
  • Multimodal fusion: Dual encoders such as CLIP handle image-text interaction through a dot product, which is effective for zero-shot classification and retrieval but insufficient for captioning and visual question answering.These limitations motivate pre-training methods for deeper multimodal fusion.
  • Multimodal fusion: Early fusion models use object detectors and treat visual features as soft prompts for a multimodal transformer.UNITER exemplifies this detector-based approach, whereas later end-to-end models directly process image patches and text tokens.
  • Multimodal language models: End-to-end pre-training has become mainstream, with models using CNNs or vision transformers and recent multimodal language models adopting LLM-inspired training.SimVLM uses a PrefixLM loss, followed by models such as Flamingo, GIT, PaLI, and PaLI-X.
  • Fine-grained understanding: GLIP reformulates object detection as phrase grounding and performs grounded language-image pre-training, while SAM targets pixel-level visual pre-training.These methods extend multimodal foundation models toward region-level and pixel-level understanding.

Visual Generation

This section surveys text-to-image generation and alignment methods that improve spatial control, prompt adherence, image editing, and concept customization. It highlights a broader shift toward unified alignment solutions that integrate multiple conditions and reduce problem-specific adjustments.

  • Text-to-Image Generation: Text-to-image generation targets images that combine high visual quality with semantic correspondence to input text.The surveyed alignment literature addresses weaknesses that prevent vanilla models from accurately following human intents.
  • Text-to-Image Generation: Stable Diffusion illustrates diffusion-based T2I generation through an image VAE, denoising U-Net, and condition encoder.Its VAE compresses 512×512 RGB images into 64×64×4 latent representations, substantially reducing the representation size.
  • Spatial Controllable Generation: Region-controlled generation improves alignment by adding spatial conditions that specify object locations, counts, relationships, and attributes.ReCo incorporates position tokens into the text interface, while ControlNet adds image-like dense conditions such as masks, edges, and depth maps.
  • Spatial Controllable Generation: Unified and disentangled control models extend dense conditioning to multiple input types and support high-fidelity foreground, background, and motion control.Uni-ControlNet and UniControl unify condition types, while Disco separates human pose and background conditions into distinct branches for dancing videos.
  • Text-based Editing: Text-based editing has progressed from mask-dependent object editing to editing synthetically generated images and following general text instructions.InstructPix2Pix adapts T2I models to accept an image and a text editing instruction, while cross-attention manipulation supports word swaps, phrase additions, and attention re-weighting.
  • Prompt Following and Unified Alignment: Alignment tuning uses inference-time guidance or reward-based finetuning to improve prompt adherence and move toward unified human-intent alignment.DDPO derives a similarity reward by comparing generated-image captions with input prompts; the survey envisions a later stage that also addresses harmful content and diverse image-text scenarios.

Unified Vision Models

This chapter examines the unification of vision models, covering efforts to create general-purpose systems and techniques for open-set recognition, task-granularity unification, and promptable interfaces.

  • The chapter surveys challenges and recent efforts toward unified vision models.
  • 4.2 From Closed-Set to Open-Set Models: It covers transforming closed-set models into open-set models.
  • 4.3 From Task-Specific Models to Generic Models: It discusses unifying vision tasks across different granularities.
  • It also examines more promptable interfaces for vision.

4.1 Overview

Unified vision models seek to address the diversity of computer vision tasks by combining open-set recognition, multi-task modeling, and LLM-like prompting into general-purpose systems.

  • Challenges: Computer vision tasks differ in inputs, including images, videos, and multimodal data, and in required output granularities.
  • Towards a unified vision model: The field is pursuing general-purpose unified vision systems despite these task-unification challenges.
  • Bridging vision and language: CLIP-style vision-language models extend closed-set classification to open-world recognition through zero-shot transfer.
  • Unified multi-task modeling: Unified multi-task modeling aims to perform well across many vision tasks instead of developing a separate model for each new task.
  • LLM-like promptable interface: LLM-like promptable interfaces use prompts to align vision-model outputs with user intents without changing model parameters.

4.2 From Closed-Set to Open-Set Models

The transition from closed-set to open-set vision models replaces fixed label vocabularies with language-based concepts, enabling recognition and localization beyond training categories.

  • Traditional recognition maps visual data to labels from a predefined closed set, limiting transfer when the vocabulary is insufficient.
  • CLIP uses a text encoder to represent textual concepts in a shared visual-semantic space, supporting open-set and zero-shot recognition.
  • Open-set models have expanded from image-level tasks to region-level localization and pixel-level understanding using large-scale image-text pairs.
  • Model initialization: Many open-set models use CLIP initialization, while others learn visual-semantic spaces from supervised models or from scratch.
  • Model design: Two-stage models decouple localization and recognition, typically combining proposal networks with a pretrained CLIP model.
  • Model pre-training: Open-set training methods include supervised learning that converts label supervision into language supervision.
  • Grounding models accept text and images to localize noun-phrase-referenced objects, with M-DETR using 1.3M labeled text-image pairs for pretraining.
  • Unified Segmentation: Unified segmentation frameworks such as X-Decoder reformulate referring segmentation as conditioned panoptic segmentation.

4.3 From Task-Specific Models to Generic Models

Generic vision models address fragmented tasks and uneven data scales by unifying inputs and outputs or by sharing functionality across tasks.

  • Specialized vision models hinder synergy across tasks because vision tasks span different spaces, times, modalities, and granularities.
  • Building unified models is also difficult because human annotations vary substantially in availability, scale, and semantics across tasks.
  • Transformers have encouraged unified models that learn from and apply to diverse vision and vision-language tasks.
  • I/O Unification: I/O unification tokenizes heterogeneous inputs and outputs into coherent visual or text-token sequences for sequence-to-sequence modeling.
  • Functionality Unification: Functionality unification uses coherent encoder-decoder architectures to accommodate multiple tasks without task-specific or modality-specific tokenizers.
  • I/O unification produces token sequences decoded by task-specific decoders, whereas functionality unification produces heterogeneous outputs combined into task-specific results.

Sparse and discrete outputs

Sparse and discrete outputs can be unified by representing task outputs as token sequences, allowing one decoder to handle textual, spatial, and multimodal tasks.

  • Sparse and discrete outputs: Language tokenizers such as BPE support I/O unification for sparse or discrete vision-task outputs.Spatial outputs can be converted into numeric-coordinate sequences and tokenized into discrete tokens.
  • UniTab: UniTab represents box coordinates numerically within special <obj> tokens to unify captioning, grounding, localization, and VQA.Its training uses pre-training, multi-task finetuning, and task-specific finetuning.
  • Pix2SeqV2: Pix2SeqV2 represents objects as [ymin, xmin, ymax, xmax, text] and uses task prompts to unify referring segmentation and keypoint detection.Its design extends sequence-based object representation to multiple task types and locations.
  • LLM-augmented: VisionLLM explores an LLM-based generic decoding interface that bridges vision systems with language models through sequential decoding.The figure describes this bridge as a sequential decoding process.
  • Visual tokenization: VQ-VAE and VQ-GAN illustrate discrete image representation approaches used to support token-based visual decoding.VQ-GAN adds adversarial and perceptual objectives to balance image quality with compression efficiency.

Dense and continuous outputs

Dense and continuous outputs are harder to express as discrete tokens, motivating dense decoders and image-oriented representations for unified vision tasks.

  • Output representations: Dense outputs such as segmentation, depth, inpainting, and editing are difficult to convert into discrete tokens because they occupy high-dimensional continuous spaces.Segmentation masks can be approximated by polygons, but most dense outputs require image-oriented decoding.
  • VQ-GAN: VQ-GAN uses patch-level discrimination and perceptual loss to preserve high-resolution image quality while maintaining a high compression rate.The discriminator regularizes image decoding at the patch level.
  • UViM: UViM unifies panoptic segmentation, depth estimation, and colorization through a two-stage dense decoding process involving a base encoder-decoder and a restricted oracle.The first stage learns output prediction and oracle-code decoding before the subsequent unified process.
  • Unified-IO: Unified-IO trains task-specific VQ-VAE models before jointly training a transformer encoder-decoder across 90 datasets.It also uses a language decoder for organic and numeric text coordinate outputs.
  • Diffusion-augmented: Prompt Diffusion initializes from Stable Diffusion and ControlNet to train an in-context image-to-image model across six vision-language tasks.The tasks include segmentation and depth estimation.
  • Functionality unification: Functionality unification groups tasks by location, semantic, and pixel-level output characteristics rather than treating them as fully isolated or aligned.Detection and phrase grounding share localization outputs, while generic and referring segmentation share masks.

Multi-task learning

Multi-task learning evolved from task-specific architectural modules toward unified models that seek greater synergy across vision and vision-language tasks.

  • Early approaches: Early multi-task learning methods explored unified architectures for different vision and vision-language tasks.These approaches established the use of shared structures for multiple tasks.
  • Vision models: Cross-stitch Networks adapt CNN layers to different tasks, but the best architecture depends on the selected tasks and generalizes poorly to new ones.The approach splits different numbers of layers from the top of CNNs for task adaptation.
  • Vision models: UberNet uses one universal CNN with routing mechanisms to support seven vision tasks while controlling memory and computation.Its architecture is designed to save memory and computing cost.
  • Multi-modal models: 12in1 combines 12 vision-language tasks in one BERT-based architecture using task-specific heads and a shared ViLBERT trunk.The reported results describe substantial improvements over single-task learning alongside reduced model requirements.
  • From modules to synergy: Earlier unified models relied on task-specific modules or heads and therefore could hardly achieve synergy across tasks.Later work instead aims to exploit relationships among vision and multimodal tasks.
  • Functionality unification: X-Decoder unifies vision and vision-language tasks with one decoder using shared parameters and task-specific routing mechanisms.This represents a functionality-based rather than purely isolated task design.

Unified learning

Unified learning uses shared multimodal representations and general decoders to apply one model across vision and vision-language tasks.

  • Unified learning: Transformers and open-set models enable inputs from different modalities to be bound into a shared semantic space.Recent systems use one model to tackle many vision and vision-language tasks after pre-training.
  • GLIPv2: GLIPv2 broadens GLIP to support tasks such as grounded captioning and visual question answering through integrated localization and vision-language pre-training.Its pre-training includes phrase grounding and region-word contrastive learning.
  • X-Decoder: X-Decoder combines multiscale image features, textual-query embeddings, and latent queries in a decoder that predicts varied task outputs.Its encoder-decoder design is intended to generalize across vision and vision-language tasks.
  • Uni-Perceiver-v2: Uni-Perceiver-v2 uses vision and text encoders with a general decoder, while a region proposal network supplies box and mask queries.The model introduces unified maximum-likelihood training for datasets at different levels.

4.4 From Static to Promptable Models

This section traces the shift from static vision models toward promptable and in-context systems that accept richer user guidance. The surveyed approaches unify diverse prompts, tasks, and visual inputs to make vision models more interactive and configurable.

  • 4.4.1 Multi-modal Prompting: Language prompts alone may be insufficient for visual interaction, motivating spatial, visual, and multimodal prompts that resolve ambiguity.These prompts complement language with information grounded in locations, examples, or other modalities.
  • 4.4.1 Multi-modal Prompting: Spatial prompting modulates vision models with location clues such as points, boxes, or arbitrary strokes.The approach reflects vision’s spatial grounding in the physical world.
  • 4.4.1 Multi-modal Prompting: SAM accepts point or box prompts and segments images at arbitrary granularity, providing a foundation for interactive segmentation applications.SAM also develops a data annotation engine to scale mask-annotated data.
  • 4.4.1 Multi-modal Prompting: SEEM supports points, boxes, and strokes on target or reference images, extracting prompt features through a visual sampler.Its visual sampler selects image features according to user-specified locations.
  • 4.4.1 Multi-modal Prompting: Other visual-prompting systems use one-shot source image–mask inputs, unified image-and-annotation prompts, or off-the-shelf vision models for diverse tasks.PerSAM personalizes segmentation, Painter covers tasks such as depth estimation and segmentation, and Prismer combines multiple vision models.
  • 4.4.2 In-context Prompting: Visual inpainting teaches models to predict dense outputs such as edges, masks, and depths from an image and binary mask.Bar et al. use discrete tokens from a pre-trained VQ-GAN and a ViT encoder to predict masked regions.
  • 4.4.2 In-context Prompting: Painter extends visual in-context learning across datasets by representing task outputs in image space and predicting continuous pixels.For segmentation, different colors encode individual instances.
  • 4.4.2 In-context Prompting: Hummingbird aggregates information across target and source images by nearest-neighbor attention over feature locations, rather than masked modeling.Its inputs include multiple prompt images with semantic label maps for dense prediction.

4.5 Summary and Discussion

The chapter surveys efforts toward open-world, unified, and interactive vision models while emphasizing intrinsic differences between vision and language. It closes by identifying unresolved challenges in data coverage, scaling, and the balance between vision- and language-centric modeling.

  • Summary and Discussion: Vision differs from language through raw physical-world signals, costly annotation, heterogeneous data and tasks, and higher storage demands.These differences complicate tokenization, knowledge conveyance, and scaling relative to language models.
  • Summary and Discussion: Figure 4.19 summarizes efforts to unify vision models from different aspects in pursuit of more intelligent vision systems.The chapter frames these efforts as a clear movement toward open-world, unified, and interactive models.
  • Summary and Discussion: Current vision training data may not cover the physical world, leaving significant challenges in novel and long-tail scenarios.The chapter identifies computer vision in the wild as an open challenge despite progress on open-set models.
  • Summary and Discussion: It remains unclear how vision models should scale and whether larger vision systems, integrated language models, or moderate vision models with LLMs are preferable.The chapter presents both scaling laws and the vision-centric versus language-centric balance as unresolved questions.
  • Summary and Discussion: The survey concludes that intelligent vision systems remain distant despite substantial progress, and aims to provide an overview that inspires next-generation model development.The conclusion preserves a qualified assessment: the field is close yet still far from human-like visual intelligence.

Large Multimodal Models: Training with LLM

Large multimodal models are moving from broad pre-training toward instruction-following systems that use LLMs to handle diverse visual-language tasks. The reviewed case studies show capabilities including in-context transfer, strong benchmark performance, zero-shot OCR, and domain-specific applications, while evaluation and the gap to proprietary systems remain open questions.

  • Multimodal in-context learning: Flamingo demonstrates multimodal in-context learning by transferring from a few image-text examples to unseen tasks without additional training.Examples include visual math problems and structured animal-information queries.
  • Instruction following and alignment: Instruction-following exposes a gap between earlier LMMs and multimodal GPT-4, motivating alignment research for multimodal systems.The cited discussion contrasts multimodal input alone with instruction-following and alignment with human intent.
  • Open-source multimodal assistants: LLaVA uses open-source resources and self-instruct data to prototype multimodal GPT-4-style interaction, reaching 85.1% and 73.5% relative scores on two LLaVA-Bench settings.The scores are reported for LLaVA-Bench (COCO) and LLaVA-Bench (In-the-Wild), respectively; the ground-truth answers were manually rewritten from GPT-4 outputs.
  • Open-source multimodal assistants: LLaVA combined with GPT-4 as a judge reaches 92.53% ScienceQA accuracy, compared with 90.92% for LLaVA alone.The scheme predicts the final answer from GPT-4’s previous answers and LLaVA’s answers.
  • Multitask instruction tuning: Instruction-tuning datasets increasingly combine conversational and academic tasks: InstructBLIP converts 26 datasets into instruction format and reports state-of-the-art zero-shot results on 13 held-out datasets.The passage reports substantial outperformance over BLIP-2 and larger Flamingo models.
  • Emergent capabilities and evaluation: LMMs can show zero-shot OCR transfer without explicit OCR training, although supervised state-of-the-art results usually outperform zero-shot systems.Across 24 text-recognition datasets, LLaVA outperformed MiniGPT-4 on 21, while WordArt was an exception where BLIP2 surpassed supervised state of the art.
  • Open research directions: Open-source models extend multimodal assistants toward chatbots and image generation, but evaluating their broad capabilities remains an urgent challenge.The survey highlights LLaVA and MiniGPT-4 for multimodal chat, while CM3leon, Emu, and GILL extend LMMs toward end-to-end image generation.

Multimodal Agents: Chaining Tools with LLM

Multimodal agents shift from separately trained task models toward chaining LLMs with multimodal experts and tools. This approach broadens multimodal capabilities and enables extensibility without retraining, while introducing tool-selection and output-quality limitations.

  • Overview: Multimodal agents chain different multimodal experts with LLMs to address complicated multimodal understanding problems.The chapter reviews their evolution, construction, capabilities, evaluation, and applications.
  • Evolution of Modeling Paradigms: The paradigm evolves from task-specific and pre-trained models toward general-purpose assistants that combine LLMs with external tools.Earlier paradigms require data curation and model training, whereas tool chaining can use a pre-trained LLM and existing tools.
  • Multimodal Tool Using: Multimodal tool use extends language-only tool augmentation by enabling LLMs to perceive multimodal signals for reasoning and actions.Language-tool outputs naturally enter LLMs as text, but multimodal agents must handle richer inputs and outputs.
  • MM-REACT Case Study: MM-REACT demonstrates visual reasoning, spatial understanding, planning, multi-image reasoning, document understanding, open-world concepts, and video analysis.These capabilities include visual math and text reasoning, visual-conditioned jokes or memes, and video summarization.
  • Extensibility: Tool chaining supports modular enhancement by upgrading the LLM or adding tools without retraining.MM-REACT can replace ChatGPT with GPT-4, while other systems incorporate models and APIs through plug-and-play mechanisms.
  • Trade-offs and Open Problems: Instruction-tuned models offer end-to-end interpretation but require costly training, whereas untrained agents are extensible yet may select incorrect tools or receive noisy outputs.Limited instruction-tuning data can constrain capabilities such as OCR, while weak domain experts can confuse planning or reasoning.
  • Future Directions: Open questions include replacing the LLM allocator with a multimodal model, creating tools on demand, and systematically evaluating tool-use accuracy and emergent capabilities.The chapter identifies API-Bank as an early step toward systematic evaluation and discusses tool creation based on user needs.

Conclusions and Research Trends

The survey organizes multimodal foundation models into specific-purpose models and emerging general-purpose assistants, emphasizing a transition toward broader visual assistance. It concludes by identifying multimodal agency, human-intent alignment, planning, memory, tool use, and rapidly evolving research as central future directions.

  • Scope and Organization: The survey covers multimodal foundation models across image self-supervised learning, language-image contrastive learning, text-to-image generation, unified vision modeling, and visual assistants.These research directions share an overarching objective of building general-purpose models.
  • Specific-Purpose Models: Specific-purpose models pre-train on large problem-related datasets and transfer to real-world scenarios of the same problem type.The survey presents visual understanding and visual generation as two major specific-purpose topics.
  • General-Purpose Assistants: General-purpose assistants commonly combine unified architectures, unified input-output formats, and general interfaces for human interaction.Their development is inspired by LLM assistants such as ChatGPT and GPT-4.
  • Transition to General-Purpose Agents: Existing visual assistants remain preliminary relative to the broader goal of general-purpose multimodal AI agents built with foundation models.The paper is organized to demonstrate the transition from specialist multimodal models to general-purpose visual assistants.
  • Generalist Agents with Multi-Modality: Future agents are envisioned as generalists that fuse language, vision, speech, and actions, with multimodal foundation models serving to perceive and synthesize visual signals.This perspective treats multimodal foundation models as components of broader agent systems.
  • Research Trends: Future research also concerns aligning systems with human intents and integrating planning, memory, tool use, fine-grained visual perception, and longer-sequence understanding.These capabilities support complex real-world tasks and agent self-improvement from prior actions.
  • Limitations: The survey has scope limits because multimodal foundation-model research evolves rapidly and many important topics are omitted.The authors attribute omissions mainly to frequently updated research innovation.
Loading 2309.10020v1…