PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao

arXiv:2210.09263v1cs.CVcs.CL

TL;DR

Vision-language pre-training addresses multimodal learning for image-text, core vision, and video-text tasks, while challenges remain in real-world deployment and domain coverage. This paper surveys methods, progress, datasets, architectures, and advanced research topics across these categories, documenting the field’s transition toward end-to-end and open-set models.

  • Problem

    Multimodal systems need to learn from vision and language, but fixed visual concepts provide limited coverage compared with semantically richer natural language, while deployment must handle varied real-world domains and costs.

  • Method

    The paper surveys VLP methods for image-text, core vision, and video-text tasks, reviewing architectures, pre-training tasks, datasets, and advanced topics.

  • Results

    VLP became mainstream from August 2019 to August 2021, with VQA accuracy increasing from ≈71% to ≈78% as end-to-end convolutional and vision Transformer methods emerged.

  • Takeaways & Limitations

    Language-augmented vision models acquire open-set and open-vocabulary recognition with strong zero-shot transfer, while video-language research is transitioning toward end-to-end video Transformer models.

  • Takeaways & Limitations

    Most vision-language datasets use natural images, leaving domain adaptation for diagrams, tables, and charts largely unaddressed.

Abstract

from arXiv · show

This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years. We group these approaches into three categories: ($i$) VLP for image-text tasks, such as image captioning, image-text retrieval, visual question answering, and visual grounding; ($ii$) VLP for core computer vision tasks, such as (open-set) image classification, object detection, and segmentation; and ($iii$) VLP for video-text tasks, such as video captioning, video-text retrieval, and video question answering. For each category, we present a comprehensive review of state-of-the-art methods, and discuss the progress that has been made and challenges still being faced, using specific systems and models as case studies. In addition, for each category, we discuss advanced topics being actively explored in the research community, such as big foundation models, unified modeling, in-context few-shot learning, knowledge, robustness, and computer vision in the wild, to name a few.

Introduction

This survey traces vision-language research from multimodal learning goals and task-specific methods to modern vision-language pre-training. It reviews image-text, core vision, and video-text applications while discussing frontier topics and the field’s broader research direction.

  • The survey positions vision-language research as a way to learn from multimodal data for tasks such as image-text retrieval and image captioning.
  • It reviews VLP across image-text tasks, core computer vision tasks, and video-text tasks.The coverage includes captioning, retrieval, question answering, visual grounding, classification, detection, segmentation, and related video problems.
  • The paper adds in-depth discussion of frontier topics including foundation models, unified modeling, in-context few-shot learning, knowledge, multilingual learning, robustness, compression, and computer vision in the wild.
  • The survey maps the field’s systems and public releases through case studies that illustrate progress and remaining challenges.
  • The authors describe a north star of unified VLP models that perform well across tasks with fixed weights or inexpensive fine-tuning.They identify task coverage, task performance, and adaptation cost as factors for future benchmarking.

Tasks, Benchmarks, and Early Models

The section introduces image-text understanding and generation tasks, their benchmarks, and the evolution of task-specific vision-language models. It uses VQA to organize early architectures around multimodal fusion, attention, visual encoders, and feature representations.

  • Tasks and Benchmarks: VL tasks are grouped by output behavior into understanding tasks that select candidates and generation tasks that produce outputs.Image-text retrieval and VQA are understanding tasks, while image captioning is a generation task.
  • Tasks and Benchmarks: Image-text retrieval matches text queries to relevant images or image queries to grounded textual descriptions in large candidate databases.Typical text-to-image retrieval databases contain 1000–5000 images, and Recall@K is commonly reported.
  • Tasks and Benchmarks: VQA answers questions about images in multiple-choice or open-ended settings, while visual reasoning benchmarks test spatial understanding and multi-step inference.Related benchmarks cover dialogue, scene text, visually impaired users, external knowledge, and rule-based reasoning.
  • Tasks and Benchmarks: Image captioning generates free-form text and is evaluated with n-gram, semantic-content, and related text-generation metrics.Datasets commonly use single-sentence captions, whereas descriptive multi-sentence caption datasets are less common.
  • Early Models: Early VQA models progressed from simple fusion to inter-modality attention, relational reasoning, bilinear pooling, and neural module networks.Attention methods assign higher weights to image regions useful for solving the task and can jointly attend to image and text.
  • Early Models: Visual representations evolved from global CNN features to grids, salient object regions, and efficient grid features, while text representations advanced from bag-of-words to RNNs and Transformers.Grid features can match region-feature performance while offering better efficiency and easier end-to-end fine-tuning when semantic content is comparable.

VLP for Image-Text Tasks

Image-text retrieval, image captioning, and VQA are among the most widely studied vision-language tasks because they require systems to comprehend both images and text. Their importance helped motivate vision-language pre-training, inspired by language-model pre-training.

  • Image-Text Tasks: Image-text retrieval, image captioning, and VQA are three of the most widely studied image-text tasks.These tasks require an AI system to comprehend both input image and text contents.
  • Image-Text Tasks: Vision-language pre-training is motivated by language-model pre-training and aims to learn transferable visual and vision-language representations.The survey presents VLP as a growing area connecting computer vision and natural language processing.
  • Image-Text Tasks: The survey examines these image-text tasks alongside related applications using modern vision-language pre-training methods.The supplied passage frames image-text tasks as a central part of the VLP research landscape.

3.1 Overview of VLP Models

Modern VLP methods are organized into dual-encoder and fusion-encoder architectures, with this chapter emphasizing fusion encoders and their evolution from detector-based to end-to-end training. Large-scale pre-training and multimodal foundation models have driven substantial VQA progress.

  • Model categories: VLP methods are broadly divided into dual encoders and fusion encoders.Dual encoders handle modality interaction through similarity, whereas fusion encoders model deeper image-text interaction.
  • Chapter scope: This chapter focuses on fusion-encoder methods and postpones detailed discussion of dual encoders to Chapter 4.
  • Architectural evolution: Early fusion encoders used pre-trained object detectors, while newer end-to-end methods back-propagate gradients into CNN or ViT vision backbones.End-to-end VLP methods have achieved new state of the art on major vision-language tasks.
  • Research progress: VQA accuracy rose from ≈66% to ≈71% between August 2017 and August 2019 as task-specific methods evolved.
  • Research progress: VLP became mainstream from August 2019 to August 2021, with OD-based methods boosting VQA accuracy from ≈71% to ≈78%.Later end-to-end methods based on convolutional networks and vision Transformers dominated the field.
  • Research progress: Scaling multimodal foundation models and pre-training datasets further boosted VQA performance from ≈80% to ≈84% between August 2021 and August 2022.Representative models include SimVLM, Florence, Flamingo, CoCa, GIT, and BEiT-3.

3.2 Model Architectures

A VLP system encodes text and visual inputs, fuses their representations, and optionally decodes multimodal features into outputs. Architectures vary by vision encoder, fusion mechanism, and encoder-only versus encoder-decoder design.

  • General framework: Given an image-text pair, separate encoders produce text and visual feature sequences that a fusion module converts into cross-modal representations.A decoder may then generate final outputs.
  • Vision encoders: Vision encoders use object detectors, CNNs, or vision Transformers, representing each image as a set of feature vectors.ViTs obtain features from linearly projected image patches with positional embeddings and a special token.
  • Multimodal fusion: Merged attention concatenates visual and text features into one Transformer, whereas co-attention processes them separately and connects them through cross-attention.
  • Multimodal fusion: For region-based models, merged and co-attention achieve comparable performance, while merged attention is more parameter-efficient.
  • Multimodal fusion: For end-to-end VLP models, co-attention performs better in METER, but there is no conclusive choice between the fusion designs.The choice remains largely empirical.
  • Output architectures: Encoder-only models fit understanding tasks such as VQA, while encoder-decoder models feed cross-modal representations into a decoder before output generation.Encoder-only models can also generate captions token by token using a causal mask.

3.3 Pre-training Objectives

VLP commonly combines masked language modeling, image-text matching, contrastive learning, and masked image modeling, with additional objectives targeting alignment and grounding. The usefulness of each objective depends on the model and setting.

  • Core objectives: Masked language modeling predicts randomly masked words from surrounding text and the paired image, typically masking 15% of input words.Seq-MLM and Prefix-LM adapt masking or attention patterns for caption generation.
  • Core objectives: Image-text matching classifies whether an image-caption pair is matched or mismatched using a global cross-modal representation.
  • Contrastive learning: Image-text contrastive learning compares image and text representations and became widely used in end-to-end VLP after CLIP and ALIGN.It was later adopted for fusion-encoder pre-training, including ALBEF.
  • Masked image modeling: Masked image modeling reconstructs masked patches or regions from visible visual features and accompanying words.Designs include regression of masked region features, discrete VQ-token reconstruction, and in-batch-negative prediction.
  • Objective choices: Recent state-of-the-art models often omit masked image modeling, and ViLT and METER report that it does not help downstream performance.Other recent methods continue to study masked vision-language modeling.
  • Fine-grained objectives: Additional objectives include optimal-transport word-region alignment and bounding-box prediction for object detection and phrase grounding.

3.4 Pre-training Datasets

VLP datasets range from academic image-caption collections to massive web-crawled corpora, whose scale supplies diverse and long-tailed concepts. Web data also introduces noisy supervision and harmful or inappropriate content that requires responsible data practices.

  • Dataset settings: Academic VLP commonly uses COCO, Visual Genome, Conceptual Captions, and SBU Captions, while industrial systems rely on much larger web-crawled datasets.
  • Industrial datasets: CLIP uses 400 million image-text pairs, ALIGN uses 1.8 billion, and WIT contains 37.6 million texts across 108 languages.
  • Industrial datasets: LAION-400M/5B provides 400 million or 5 billion publicly released image-text pairs filtered using CLIP cosine similarity above a threshold of 0.3.
  • Data quality: Web-crawled texts may be descriptive, malformed, very short, or unrelated to image content, creating noisy supervision for VLP.
  • Data quality: Web-collected data contains diverse fine-grained and long-tailed concepts, providing richer training resources than human-annotated datasets.
  • Data guidance: Audits of large web datasets found pornographic imagery, racist slurs, and harmful stereotypes, so model development should include transparent, responsible data and model documentation.

3.5 Advanced Topics

The survey reviews advanced VLP directions including large-scale and unified models, few-shot learning, knowledge, multilingual modeling, and robustness. It highlights contrasting pre-training strategies, emerging capabilities, and remaining limitations in visual and multimodal representation.

  • Advanced VLP research spans big models, unified modeling, in-context few-shot learning, knowledge, multilingual learning, robustness, compression, and computer vision in the wild.
  • Big Models: Most big VLP models use contrastive pre-training, generative pre-training, or both, combining retrieval and open-set classification with multimodal generation and understanding.ITC supports fast retrieval and open-set classification, while MLM or LM after fusion supports captioning and VQA.
  • Big Models: Roughly 1B-parameter models are commonly pre-trained on roughly 1B-10B image-text pairs.
  • Big Models: BEiT-3 achieves state-of-the-art VQA and other VL-task performance using only masked data modeling and a multi-way Transformer.
  • In-Context Few-Shot Learning: Few-shot VLP transfers language-model in-context learning to multimodal settings through image-text alignment, caption-based prompting, or frozen vision-language components.Frozen uses an image encoder aligned through image captioning; PICa prompts GPT-3 with image captions; Flamingo bridges frozen encoders with gated cross-attention.
  • In-Context Few-Shot Learning: 4-shot PICa prompting outperforms supervised SoTA on OK-VQA, but gains on VQAv2 are limited because captions lose fine-grained visual information.
  • Multilingual VLP: Multilingual VLP includes CCLM and PaLI, with CCLM claiming zero-shot cross-lingual transfer beyond translate-test performance of representative English VL models.

3.6 Text-to-Image Generation

Text-to-image generation has evolved from early variational and GAN-based systems toward large-scale autoregressive and diffusion-based VLP models. These approaches now support high-fidelity synthesis, while related multimodal models extend across image and text generation tasks.

  • Text-to-image generation produces images reflecting textual descriptions and can be viewed as the inverse of image captioning.
  • VLP-based text-to-image methods fall into VQ-token-based autoregressive models and diffusion-based models.Representative systems include DALL-E and Parti for autoregressive modeling, and DALL-E 2 and Imagen for diffusion.
  • Auto-regressive Modeling: Large-scale autoregressive Transformers trained on image-text pairs produce high-fidelity, controllable synthesis through text prompts.DALL-E demonstrates this route, while Parti scales a similar Transformer encoder-decoder architecture.
  • Bi-Directional Image-Text Generation: Joint generative pre-training over text and VQ-VAE image tokens supports diverse downstream tasks including style learning, super-resolution, captioning, and retrieval.
  • Diffusion-based Methods: DALL-E 2, Imagen, and Stable Diffusion further advance diffusion-based generation, with Latent Diffusion operating in continuous latent space rather than pixel space.
  • Text-to-Video Generation: Text-to-video systems such as Make-A-Video, Imagen Video, and Phenaki have significantly improved the quality of text-to-video generation.

VLP for Core Vision Tasks

VLP reframes core vision recognition as image-text retrieval, enabling open-set recognition and richer language-based concepts. The section reviews representative methods for image classification, object detection, and segmentation, while highlighting transferability and evaluation challenges.

  • Overview: Open-set recognition is enabled by reformulating classification as image-to-text retrieval.The retrieval formulation replaces a fixed category set with image-language matching.
  • Overview: Natural language represents a larger and semantically richer concept pool than fixed classification labels.Text sequences can also represent external knowledge beyond compact category heads.
  • Overview: The chapter reviews UniCL, CLIP, and ALIGN for image classification; ViLD, RegionCLIP, and GLIP for detection; and LSeg, OpenSeg, and DenseCLIP for segmentation.These case studies illustrate the shift toward language-supervised core vision tasks.
  • Image Classification: UniCL uses bidirectional supervised contrastive learning that treats samples sharing a language concept as positives.Scaling UniCL to 800M training samples led to Microsoft Florence, which achieved state-of-the-art performance on many tasks at that time.
  • Image Classification: CLIP and ALIGN assume one-to-one image-caption mappings and treat only diagonal batch pairs as positive.Their formulation is best suited to image-text batches without duplication in either modality.
  • Trends: The Image Classification in the Wild benchmark addresses incomparability caused by evaluating methods on different downstream datasets.It formalizes task-level evaluation with 20 public datasets, while adaptation cost is measured through sample- and parameter-efficiency.

VLP for Video-Text Tasks

This chapter systematically reviews VLP for video-text tasks, covering task types, model architectures, representative methods, and the temporal demands of video understanding.

  • Overview: Video-text VLP requires modeling spatial information within frames and temporal dependencies across frames.The chapter organizes its review around video-text tasks and the architectures used to address them.

5.1 Video-Text Tasks

Video-text tasks include retrieval, question answering, and captioning, spanning matching, grounding, classification, and natural-language generation settings.

  • Text-to-Video Retrieval: Text-to-video retrieval matches a natural-language query to a relevant video or video segment in a large corpus.Video retrieval finds an entire video, while moment retrieval grounds a query in a temporal interval.
  • Text-to-Video Retrieval: Video retrieval is evaluated with Recall@K for K=1, 5, 10, 100.Common benchmarks include MSVD, MSRVTT, LSMDC, YouCook2, VATEX, DiDeMo, and ActivityNet Captions.
  • Video Question Answering: Video question answering asks models to answer questions based on video content in multiple-choice or open-ended settings.Both settings are evaluated using accuracy, with multiple-choice QA commonly formulated as classification.
  • Video Captioning: Video captioning generates a natural-language description covering events, objects, their temporal evolution, and their relations.It is the only generation task among the three video-text tasks described here.

5.2 Model Architectures

Video-text models combine video and text encoders with multimodal fusion, using dual or fusion encoders and either direct output layers or decoders. Video encoders have evolved toward end-to-end learning to better align representations with video-text tasks.

  • General Framework: A typical video-text model extracts token and visual-feature sequences, then projects them through a multimodal fusion module.Visual features may represent frames, regions, or patches, depending on the vision encoder.
  • Encoder Designs: Dual encoders align global video and text representations with lightweight similarity operations, favoring fast text-to-video retrieval.Their shallow cross-modal interaction is less effective for video question answering.
  • Encoder Designs: Fusion encoders concatenate video and text features and use Transformer layers to capture fine-grained interactions.Deep fusion models show strong performance on video question answering and captioning while remaining competitive for retrieval.
  • Video Encoder: Video encoders must capture spatial frame content and temporal dynamics across frames.The field has progressed from multiple offline feature extractors toward a single video encoder learned end-to-end.
  • Output Architectures: Most video-text models are encoder-only, while UniVL, MV-GPT, and Support Set add a decoder between fusion and the output layer.Encoder-only models generate outputs directly from cross-modal representations.

5.3 Pre-training Tasks

Video-text pre-training combines general multimodal objectives with video-specific tasks that model cross-modal correspondence, temporal order, and masked visual content. These objectives also address noisy or weak video-text alignment.

  • General Pre-training Tasks: Fusion encoders commonly use Masked Language Modeling and Video-Text Matching to learn from paired video-text inputs.VTM is formulated as binary classification over positive and negatively constructed video-text pairs.
  • General Pre-training Tasks: Video-Text Contrastive Learning predicts matched video-text pairs from all pairings in a batch using video-to-text and text-to-video similarities.Dual encoders commonly compute these similarities with a lightweight inner product.
  • Alignment-aware Objectives: Noisy alignment between visual frames and speech-transcribed subtitles motivates multiple-instance contrastive learning, variable-length temporal pairing, and harder negative sampling.MIL-NCE uses weak narrated-video signals, while VideoCLIP constructs temporally overlapped clips and retrieves harder negatives from other videos.
  • Alignment-aware Objectives: Token-aware VTC computes contrastive loss over selected words such as nouns and verbs, then combines this signal with naive and fusion-based VTC losses.TACo uses cascade sampling to limit fusion-encoder computation to a small subset of hard negatives.
  • Video-specific Objectives: Frame Order Modeling scrambles frames and trains models to recover absolute or relative temporal order, explicitly modeling video chronology.Absolute-order FOM randomly shuffles 15% of frames; relative-order FOM predicts pairwise temporal ordering after scrambling.
  • Video-specific Objectives: Masked Video Modeling reconstructs masked visual patches, features, or discrete visual tokens using neighboring video inputs and paired text.VideoBERT uses hierarchical k-means to create discrete visual tokens and masks 15% of them for prediction.

5.4 Pre-training Datasets

Video-text datasets differ in source, duration, scale, accessibility, and caption quality. YouTube collections provide massive scale but noisy ASR text, whereas web-crawled datasets are smaller and often more precisely captioned.

  • Dataset Characteristics: Video-text datasets are harder to collect and noisier than large-scale image-text datasets, with YouTube videos typically paired with inaccurate ASR text.YouTube videos average six minutes, while web-crawled videos are usually under one minute and may use globally descriptive alt-text.
  • Representative Datasets: HD-VILA-100M provides 100M clip-sentence pairs from 3.3 million YouTube videos totaling 371.5K hours at 720p resolution.It expanded video-text data beyond the 240p or 360p resolution common in earlier datasets.
  • Representative Datasets: WebVid2.5M contains 2.5M scraped text-video pairs and uses manually generated captions that are mostly well-formed and more precise than typical noisy sources.Although more than 20x smaller than YouTube-based datasets, WebVid2.5M is described as higher quality and widely adopted.
  • Representative Datasets: The TV Dataset contains 22K clips from 925 episodes across six TV series, with 60–90-second clips covering long-range scenes and complex interactions.Its accompanying texts are human-written subtitles.
  • Dataset Characteristics: YouTube-based datasets such as HowTo100M, HD-VILA-100M, and YTTemporal range from 100M to 1B video clips, while WebVid and TV Dataset are smaller.The cited scale comparison distinguishes large YouTube collections from smaller web and television datasets.
  • Image-text Data for Video Pre-training: Researchers also transfer image-text datasets such as COCO, Visual Genome, SBU Captions, Conceptual Captions, and CC12M into video-text pre-training corpora.Images are treated as videos with temporal size 1.

5.5 Advanced Topics

Advanced video-text research extends pre-training across channels, tasks, datasets, and languages while examining whether current benchmarks measure genuine temporal understanding. The survey highlights transfer from image-text models, unified modeling, robustness, and broader evaluation as active directions.

  • Transferring Image-text Models to Video-text Tasks: Image-text pre-training improves downstream video-text performance and has supported state-of-the-art results across retrieval, question answering, and captioning.Scaling image-text models to hundreds of millions or billions of pairs is associated with strong performance on these video-text tasks.
  • Transferring Image-text Models to Video-text Tasks: Transferring image-text models can reduce video-text pre-training cost because image-text data are cleaner and easier to collect.The survey contrasts the abundance and alignment quality of image alt-text pairs with video-text data.
  • Learning from Multi-channel Videos: Most video-text models focus on visual frames despite videos also containing speech-transcribed language and environmental audio.This motivates multi-channel video encoding and benchmarks that evaluate more than single-channel visual inputs.
  • Learning from Multi-channel Videos: HERO learns local frame-subtitle alignments and global temporal context, and its performance on single-channel tasks improves substantially when ASR inputs are added.HERO can also generalize from multi-channel pre-training to single-channel video-text tasks.
  • Benchmarks and Evaluation: VALUE addresses inconsistent downstream evaluation with 11 datasets spanning retrieval, question answering, and captioning across diverse video domains.The benchmark uses multi-channel videos and provides a live leaderboard.
  • Analysis on Video-text Benchmarks/Models: Benchmark analyses find that single-frame training can improve results on many existing tasks, while retrieval evaluation may penalize relevant captions under a single-match assumption.Proposed responses include temporally challenging subsets and semantic-similarity evaluation protocols.
  • Analysis on Video-text Benchmarks/Models: Text manipulations can easily fool pre-trained video-language models, indicating reliance on spurious clues in training data.This finding motivates analysis of model weaknesses and robustness.
  • Multilingual Video-text Learning: Video-text modeling remains less explored in non-English languages because large-scale non-English pre-training and downstream datasets are scarce.Initial efforts include Chinese and multilingual video-text datasets.

VL Systems in Industry

VL capabilities are increasingly embedded in commercial products and cloud services, especially for automated image description and accessibility. Deployment still faces unresolved domain-adaptation challenges and practical serving-cost constraints.

  • Commercial Applications: Commercial products use VL systems to generate image descriptions that support accessibility and image understanding.Examples include iPhone VoiceOver, Microsoft Office, Microsoft Edge, Chrome OS, Seeing AI, and Facebook.
  • Commercial Applications: Microsoft PowerPoint calls an image-captioning service to generate editable alt text after a user selects “View Alt Text.”The example output is “A cat walking in the snow.”
  • Accessibility: Screen readers can read automatically generated image descriptions aloud, extending VL assistance to users who are blind or have low vision.This workflow is described for Microsoft Edge, Chrome OS, and Seeing AI.
  • Cloud Services: Cloud services provide image and video analysis, custom model training, image search, content moderation, face analysis, text detection, and related capabilities.Examples include Microsoft Azure Computer Vision, Google Cloud Vision AI, Amazon Rekognition, and Alibaba Cloud Image Search.
  • Deployment Challenges: Real-world deployment requires models to generalize across unpredictable domains, but domain adaptation for VL remains comparatively underexplored.The paper specifically identifies diagrams, tables, and charts as non-natural image domains that remain unaddressed.
  • Deployment Challenges: 73% smaller models can reduce inference time cost by 94% while retaining 94-97% of accuracy across multiple VL tasks.MiniVLM is presented as an example of reducing serving cost without sacrificing most task performance.

Conclusions and Research Trends

The survey synthesizes VLP progress across image-text, core vision, and video-text tasks while identifying general-purpose multimodal foundation models as a central research direction. It highlights advances in scale, task unification, open-vocabulary recognition, and efficient adaptation, alongside unresolved challenges in knowledge freshness, robustness, and complex few-shot outputs.

  • Survey Scope: The survey covers VLP for image-text tasks, core vision tasks, and video-text tasks, including their architectures, pre-training tasks, datasets, and advanced topics.The reviewed image-text methods span captioning, visual question answering, retrieval, and grounding; core vision coverage includes classification, detection, and segmentation.
  • Image-Text VLP: Image-text VLP has shifted from offline object-detector features toward end-to-end models, while training scale has grown from about 4M images to over 10B image-text pairs.The survey also discusses unified modeling, few-shot learning, knowledge, robustness, multilingual VLP, compression, and efficient adaptation.
  • Core Vision VLP: Core vision tasks can be reformulated as VL problems, giving language-augmented models open-set and open-vocabulary recognition with strong zero-shot transfer.The survey describes image classification as image retrieval and object detection as phrase grounding, and connects richer language semantics with improved generalization.
  • Video-Text VLP: Video-text VLP follows a similar transition toward end-to-end models using video Transformers instead of offline extracted video features.The survey reviews video retrieval, question answering, captioning, architectures, pre-training tasks, and datasets.
  • Unified Modeling: General-purpose multimodal foundation models require scalable unified architectures that can support image-level, region-level, pixel-level, and vision-language tasks.The survey presents unification as a route toward models that can be readily adopted across downstream tasks.
  • Model Scaling: Scaling VLP remains in an early stage relative to language-model scaling, while larger models may enable zero-shot and few-shot generalization and emergent abilities.The survey identifies models such as Florence, CoCa, Flamingo, PaLI, and BEiT-3 as examples of the scaling trend.
  • Open Challenges: In-context few-shot learning works for text-output tasks such as question answering, captioning, and classification, but localization and pixel-output tasks remain unresolved.The limitation follows from the diversity of vision tasks and their need for outputs beyond text sequences.
  • Open Challenges: The field still needs efficient adaptation, external knowledge integration, and careful robustness evaluation because models grow larger, knowledge changes, and standard benchmarks may not reflect real-world capability.The survey also notes that its coverage omits many rapidly emerging topics and methods.
Loading 2210.09263v1…