PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundational Models Defining a New Era in Vision: A Survey and Outlook

Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Fahad Shahbaz Khan

arXiv:2307.13721v1cs.CVcs.AI

TL;DR

Vision foundation models address the need for systems that can reason across visual scenes and other modalities. The survey synthesizes their architectures, training, prompting, applications, and open challenges, reporting broad coverage rather than a single benchmark result. It highlights adaptable multimodal models while emphasizing unresolved evaluation and security concerns.

  • Problem

    Vision systems need to represent compositional scenes and their relations across language and other modalities, but existing computer-vision models are often restricted to predetermined categories.

  • Method

    The survey systematically categorizes vision foundation models by architecture, training objective, prompting design, modality, downstream adaptation, and application.

  • Results

    The survey covers textually prompted, visually prompted, heterogeneous-modality, and embodied models across zero-shot recognition, localization, dialogue, cross-modal understanding, and medical applications.

  • Takeaways & Limitations

    Foundation models can serve as generalist vision systems whose outputs are adapted through textual, visual, or heterogeneous prompts without retraining.

  • Takeaways & Limitations

    Comprehensive evaluation remains difficult for conversational vision-language models because their possible visual tasks and reasoning capabilities are highly diverse.

Abstract

from arXiv · show

Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world. The complex relations between objects and their locations, ambiguities, and variations in the real-world environment can be better described in human language, naturally governed by grammatical rules and other modalities such as audio and depth. The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time. These models are referred to as foundational models. The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions. In this survey, we provide a comprehensive review of such emerging foundational models, including typical architecture designs to combine different modalities (vision, text, audio, etc), training objectives (contrastive, generative), pre-training datasets, fine-tuning mechanisms, and the common prompting patterns; textual, visual, and heterogeneous. We discuss the open challenges and research directions for foundational models in computer vision, including difficulties in their evaluations and benchmarking, gaps in their real-world understanding, limitations of their contextual understanding, biases, vulnerability to adversarial attacks, and interpretability issues. We review recent developments in this field, covering a wide range of applications of foundation models systematically and comprehensively. A comprehensive list of foundational models studied in this work is available at \url{https://github.com/awaisrauf/Awesome-CV-Foundational-Models}.

1 INTRODUCTION

Foundation models use large-scale data and multimodal learning to support adaptable vision systems, while this survey organizes their architectures, prompting strategies, applications, and open challenges.

  • Motivation: Foundation models are trained on broad, large-scale data and adapted to downstream tasks after pre-training.The recent expansion of model and data scale has accelerated their development.
  • Vision-language models: Vision-language models such as CLIP provide transferable representations and zero-shot performance for tasks including image classification and object detection.These models are typically trained on millions of web-collected image-text pairs.
  • Prompting: Visual-prompted models such as SAM perform class-agnostic segmentation from prompts including boxes, points, or masks.SAM is trained on billions of object masks and can be adapted to medical imaging, video, robotics, and remote sensing.
  • Survey scope: The survey systematically covers textually prompted, visually prompted, heterogeneous-modality, and embodied foundation models, along with architectures, objectives, training, and prompting.It further divides textual models into contrastive, generative, hybrid, and conversational categories.
  • Open challenges: The survey identifies evaluation, real-world understanding, contextual reasoning, bias, adversarial vulnerability, and interpretability as open challenges.Its related-work discussion distinguishes this broader coverage from surveys focused mainly on textual vision-language models or language models.

2 PRELIMINARIES

The survey frames computer-vision foundation models around architecture, training objectives, large-scale training, and prompting.

  • Scope of preliminaries: The survey focuses on model architecture, training objectives, and large-scale training and prompting as three main contributing factors.These factors structure its background discussion of foundation models in computer vision.

2.1 Foundational Models and Scope of the Survey

The survey defines foundation models as broadly pretrained base models that can be adapted across applications, while narrowing its scope to multimodal vision-language systems and excluding image-generative model families.

  • Definition: Foundation models are base models trained on large-scale data with self-supervised or semi-supervised learning and adapted to multiple downstream tasks.This paradigm can replace several narrow task-specific models with broader reusable bases.
  • Survey scope: The survey focuses on multimodal vision-language foundation models that support computer-vision tasks producing outputs such as text or segmentation masks.Its scope concerns models trained on large-scale data and adapted across several vision tasks.
  • Exclusions: GANs, VAEs, and diffusion models are excluded because dedicated surveys already cover them and the surveyed model class spans a broader range of downstream applications.This is an explicit scope boundary rather than a claim that those models are unimportant.

2.2 Architecture Types

The survey groups vision-language models into dual-encoder, fusion, encoder-decoder, and adapter-LLM architectures based on how visual and textual information is processed.

  • Dual-encoder: Dual-encoder architectures separately encode visual and textual inputs, then optimize their outputs with an objective function.The two modalities produce representations before alignment or other objective-based optimization.
  • Fusion: Fusion architectures combine representations from vision and text encoders through an additional fusion encoder.The fusion component learns joint multimodal representations.
  • Encoder-decoder: Encoder-decoder architectures process multimodal information through sequential encoding and decoding.The survey presents this as a distinct architecture family from parallel encoders and fusion designs.

2.3 Training Objectives

The survey organizes vision-language training objectives around contrastive losses that align paired modalities and generative losses that reconstruct or predict tokens from contextual inputs.

  • Contrastive Objectives: Image-Text Contrastive loss matches correct image-text pairs while separating incorrect configurations within a batch.It maximizes cosine similarity for N correct pairs and minimizes similarity for N^2 − N incorrect pairs.
  • Contrastive Objectives: Image-to-text and text-to-image contrastive losses are combined, with temperature τ controlling the similarity scaling.The total loss is the sum of the two directional terms.
  • Contrastive Objectives: Image-Text Matching loss predicts whether an image-text pair is positively or negatively matched using cross-entropy.A few perceptron layers predict the probability that a pair is matched.
  • Generative Objectives: Generative objectives include masked language modeling, autoregressive language modeling, captioning, and multimodal variants.The survey also lists masked multimodal, image-conditioned, image-grounded, masked-image, and parallel-prediction losses.
  • Generative Objectives: Masked language modeling reconstructs masked tokens, whereas language modeling predicts the current token from previous tokens.Captioning additionally conditions next-token prediction on previous tokens and an image.

2.4 Large-scale Training

Large-scale training and prompting are central to vision-language foundational models, which use varied datasets and adaptation settings across downstream applications.

  • Training and Adaptation: Large-scale data supports pre-training, fine-tuning, and prompting in vision-language foundational models.The survey presents these stages together as crucial components of the model pipeline.
  • Pre-training Data: Pre-training data spans image-text datasets, partially synthetic datasets, and combinations of benchmark datasets.Examples include WebImageText for CLIP, SA-1B for SAM, and PMD for FLAVA.
  • Pre-training Data: Web-scale image-text datasets are filtered from web crawls, while open curation efforts include LAION and COYO-700M.Many subsequent datasets are not public, motivating open-source curation efforts.
  • Pre-training Data: Teacher-generated mask-description data provides a cost-effective alternative when grounding datasets require expensive human annotation.GLIP first adopted this strategy, and later work scaled it to a billion examples.
  • Pre-training Data: Dataset combinations use image-text benchmarks, non-image-text datasets, and template-based prompt engineering to create training descriptions.The survey describes combinations involving captioning, visual question answering, and visual grounding-related data.
  • Fine-tuning: Fine-tuning targets task performance, specific capabilities, or instruction following across downstream vision tasks.Examples include open-world object detection, visual grounding, and InstructBLIP-style instruction tuning.
  • Prompting: Prompt engineering converts vision datasets into image-text training data and supports interaction with vision-language models on vision tasks.Template prompts such as “image of a {label}” add contextual descriptions to one-word labels.

3 TEXTUALLY PROMPTED MODELS

Textually prompted vision foundation models span contrastive, generative, and unified vision-language designs, extending global image-text representations toward pixel-, region-, and task-level understanding. The survey organizes these approaches by training objectives, modality alignment, prompting, and their ability to support diverse vision and language tasks.

  • 3.1.1 CL for General Purpose Foundational Models: CLIP jointly trains image and text encoders to align correct image-caption pairs in a shared multimodal embedding space.The encoders may use ViT or CNN vision backbones and GPT-like text transformers.
  • 3.1.1 CL for General Purpose Foundational Models: MaskCLIP combines random image masking with mean-teacher self-distillation to learn local semantic features alongside whole-image representations.The teacher produces whole-image representations while the student processes masked images.
  • 3.1.1 CL for General Purpose Foundational Models: 69.3% zero-shot ImageNet accuracy is achieved by CLIPA after four days of training on eight A100 GPUs.CLIPA exploits an inverse scaling law in which larger image-text models can use smaller token sizes with limited accuracy loss.
  • 3.1.2 CL for Visual Grounding Foundational Models: CLIP variants address localization weaknesses through pixel- and region-level adaptations such as MaskCLIP, CRIS, and GLIP.These methods introduce masked self-distillation, pixel-level learning, or phrase-region alignment to extend beyond global features.
  • 3.2 Generative Learning: Generative vision-language models support few-shot and low-data applications by conditioning language generation on visual representations and text.Frozen concatenates textual and visual embeddings for autoregressive decoding, while Flamingo uses Perceiver Resample to handle variable-sized image and video inputs.
  • 3.3.1 Foundational Models for Generic Vision-Language Learning: Unified models reformulate multiple vision tasks as multimodal conditional text generation or combine unimodal and multimodal encoders within one architecture.UNITER, VL-x, FLAVA, BridgeTower, PaLI, and related systems differ in how they connect modality-specific encoders and multimodal components.
  • 3.3.1 Foundational Models for Generic Vision-Language Learning: CoCa combines contrastively trained unimodal encoders with a decoupled multimodal decoder, supporting encoder, dual-encoder, and encoder-decoder capabilities.Its contrastive objective learns aligned global representations while the decoder provides multimodal generation.
  • 3.3.1 Foundational Models for Generic Vision-Language Learning: 86.3%, 88.0%, and 91.0% ImageNet accuracy are reported for CoCa under zero-shot, few-shot, and light fine-tuning settings, respectively.The results illustrate performance across progressively more task-specific adaptation regimes.

4 CONVERSATIONAL VISION-LANGUAGE MODELS

Conversational vision-language models extend language models with visual inputs to support dialogue, description, reasoning, and instruction following. The reviewed approaches align visual features with language models using projected visual tokens, adapter prompts, curated instruction data, or specialized video and radiology encoders.

  • Overview: Conversational VLMs combine multimodal inputs with language models to support human-like conversations about visual content.The section distinguishes these models as textually prompted systems designed for multimodal dialogue.
  • Closed-source conversational models: GPT4 supports multimodal conversations, intricate image descriptions, and complex real-world problem solving, but its architecture is undisclosed and access is provided through a paid API.GPT4 is described as transformer-based, pretrained with next-token prediction, and fine-tuned using RLHF.
  • Open-source conversational models: MiniGPT-4 combines Vicuna with ViT-G and a Q-Former, trains a projection layer through two stages, and exhibits GPT4-like visual description and explanation abilities.Training uses large multimodal datasets followed by curated instruction-image-text pairs.
  • Instruction-following models: LLaVA uses ChatGPT- and GPT4-curated multimodal instruction data with LLaMA and CLIP, training first for alignment and then end-to-end fine-tuning.The model is reported to explain complex visual scenes, while Video-ChatGPT and XrayGPT adapt related designs to video and radiograph dialogue.
  • Instruction-following models: LLaMA-Adapter injects learnable adaptation prompts and CLIP-derived visual tokens into LLaMA, while its V2 variant adds early visual fusion, disjoint training, and a visual expert.V2 improves instruction following, conventional vision-language tasks, and visual instruction following compared with V1.
  • Grounded conversation: Conversational VLMs generally reason about holistic images rather than visual grounding, motivating efforts toward conversations focused on particular objects.The survey identifies visually grounded dialogue as an extension of conversational vision-language modeling.

5 VISUALLY PROMPTED MODELS

Visually prompted foundation models extend segmentation beyond fixed, retrained systems by accepting text, points, boxes, masks, or images as prompts. The section covers general segmentation models, interactive multimodal prompting, domain adaptation, tracking, captioning, and mobile deployment.

  • 5 VISUALLY PROMPTED MODELS: Visually prompted models support segmentation through diverse inputs, including text, points, bounding boxes, and masks.The surveyed models target generalized segmentation rather than a single specialized segmentation type.
  • 5.1 Foundational Models for Segmentation: CLIPSeg conditions a transformer decoder on joint text-visual CLIP embeddings, enabling segmentation from arbitrary text or image prompts at test time.Its CLIP-based image and text encoders provide the representations used by the decoder.
  • 5.1 Foundational Models for Segmentation: SegGPT frames diverse segmentation datasets and task types as an in-context learning problem for one generalizable foundational model.The approach addresses the challenge of accommodating part, semantic, instance, panoptic, person, medical, and aerial segmentation within one framework.
  • 5.1 Foundational Models for Segmentation: SAM is a zero-shot segmentation model trained from scratch on 1.1 billion masks and 11 million images, combining image and prompt embeddings with a lightweight mask decoder.Its prompts can be boxes, points, text, or masks, including ambiguous prompts that require a valid segmentation mask.
  • 5.1 Foundational Models for Segmentation: SEEM extends visual prompting toward a multimodal interface intended to support interactive segmentation through human conversations and high-level semantic tasks.It is motivated by limitations in SAM’s interaction options and semantic-task support.
  • 5.1.1 SAM for Medical Segmentation: SAM adaptations address medical segmentation, tracking, remote sensing, captioning, and other domains, but natural-image training makes medical generalization difficult.MedSAM fine-tunes SAM’s decoder on over 200,000 masks across 33 segmentation tasks and 11 modalities while freezing the image and prompt encoders.
  • 5.1.2 SAM for Tracking: SAM-PT uses sparse point tracking for video segmentation, requiring only sparse point annotation on the first frame to identify the target object.It tracks positive and negative points and uses point trajectories to initialize segmentation across video frames.
  • 5.1.4 SAM for Captioning: Caption AnyThing combines visual prompts, SAM masks, image captioners, and a language-model refiner to produce region-focused captions in user-specified styles.The visual prompt selects a region of interest, while language controls tailor the resulting description.

6 HETEROGENEOUS MODALITIES BASED MODELS

Heterogeneous-modality models align image, video, audio, depth, thermal, and inertial signals with language or shared representations. The surveyed systems extend CLIP, concatenate multimodal temporal inputs, or connect visual encodings to multimodal language models for retrieval, generation, and reasoning.

  • 6 HETEROGENEOUS MODALITIES BASED MODELS: CLIP2Video and AudioCLIP extend CLIP with video and audio, respectively, to model spatial-temporal or tri-modal relationships.AudioCLIP adds text-to-audio and image-to-audio losses alongside text-to-image similarity.
  • 6 HETEROGENEOUS MODALITIES BASED MODELS: ImageBind learns shared representations across paired modalities and expands zero-shot capabilities to audio, depth, thermal, and IMU readings.Its design aligns visual features with multiple sensory experiences associated with images.
  • 6 HETEROGENEOUS MODALITIES BASED MODELS: COSA uses concatenated video-text samples with sequential frame captions to support visual retrieval, captioning, question answering, and both discriminative and generative tasks.The concatenated corpus is described as providing richer scene transformations, reduced visual redundancy, and finer-grained frame descriptions than short-form video-text data.
  • 6 HETEROGENEOUS MODALITIES BASED MODELS: Valley bridges video, image, and language with a projection module, multilingual instruction tuning, and spatiotemporal pooling for unified visual encoding.The model is presented as a multimodal framework for video-related tasks.

7 EMBODIED FOUNDATIONAL AGENTS

Embodied foundational agents connect language and multimodal inputs to robot manipulation and open-ended environments. The section covers multimodal task prompting, systematic generalization, and lifelong learning through data-rich interactive worlds.

  • 7 EMBODIED FOUNDATIONAL AGENTS: Embodied agents connect textual representations to real-world visual and physical sensor modalities for robot manipulation and grounded decision-making.The surveyed agents use continuous sensor inputs and natural-language task specifications.
  • 7 EMBODIED FOUNDATIONAL AGENTS: ViMA interleaves textual and visual tokens as multimodal prompts to express a wide range of robot manipulation tasks.Its evaluation includes a four-level protocol for systematic generalization.
  • 7 EMBODIED FOUNDATIONAL AGENTS: MineDojo provides a natural-language API for open-ended Minecraft tasks using data from 30K+ YouTube videos, 6K+ Wiki pages, and 340K+ Reddit posts.The environment supports specifying tasks, changing world settings, and observing and acting within Minecraft.
  • 7 EMBODIED FOUNDATIONAL AGENTS: VOYAGER is an LLM-powered lifelong-learning agent designed to explore Minecraft, develop skills, adapt to feedback, and seek new tasks.Its design is motivated by human-like lifelong learning and exploration.

8 OPEN CHALLENGES & RESEARCH DIRECTIONS

The survey identifies shared challenges for vision-language foundation models, including limited openness, difficult evaluation, data and compute demands, security vulnerabilities, and weak grounding in physical reality.

  • Multimodal Open-source Models: Matching GPT4's capabilities with open-source multimodal models remains a major challenge.The survey notes that GPT4 is closed-source and its training details remain unknown, while public models have not yet matched its capabilities.
  • Evaluation and Benchmarking: Open-ended conversational vision-language models are difficult to evaluate comprehensively because visual tasks and reasoning capabilities are diverse.The survey discusses LLM-as-a-judge evaluations but notes that using an external GPT4 model as a gold standard remains debatable.
  • Large Data and Compute Requirements: Training and inference are data- and compute-intensive, limiting accessibility and scalability in real-world applications.Large-scale labeled data can be costly to acquire, while many parameters make inference expensive.
  • Security and Robustness: Foundational models face vulnerabilities from indirect prompt injection, adversarial inputs, altered in-context exemplars, and corrupted visual prompts.Reported consequences include manipulated content, fraud, malware, intrusion, personal-information leakage, and denial of service.
  • Lack of Real-world Understanding: Language and vision foundation models lack deep real-world understanding because their training differs from human learning and reasoning grounded in physical reality.The survey calls for world models and alignment of heterogeneous modalities in egocentric perception and embodied AI.

9 CONCLUSION

The survey reviews vision and language foundation models, organizing them by architectures, training objectives, adaptation, and prompting designs while identifying major open challenges.

  • 9 CONCLUSION: The survey categorizes foundation models as textually prompted, visually prompted, or based on heterogeneous modalities.It focuses on architecture types, training objectives, downstream task adaptation, and prompting designs.
Loading 2307.13721v1…