PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language Models for Vision Tasks: A Survey

Jingyi Zhang, Jiaxing Huang, Sheng Jin, Shijian Lu

arXiv:2304.00685v2cs.CV

TL;DR

Visual recognition research faces costly labelled-data collection and task-specific training requirements. This survey systematically reviews VLM architectures, objectives, datasets, methods, transfer and distillation approaches, benchmarks, and research directions. It reports strong zero-shot classification performance while identifying scaling, computation, and fair-benchmarking limitations.

  • Problem

    Visual recognition relies on large-scale crowd-labelled data and separate task-specific DNN training, motivating approaches that reduce these requirements.

  • Method

    The paper systematically surveys VLM visual recognition research across foundations, datasets, pre-training, transfer learning, knowledge distillation, benchmarking, and future directions.

  • Results

    VLM pre-training achieves remarkable zero-shot prediction across image classification tasks, while dense visual recognition remains less developed.

  • Takeaways & Limitations

    The survey's comparative summaries provide a broad view of VLM development and identify directions for future visual recognition research.

  • Takeaways & Limitations

    VLM performance can saturate with scaling, while large-scale pre-training and large models require substantial computation, memory, and resources.

Abstract

from arXiv · show

Most visual recognition studies rely heavily on crowd-labelled data in deep neural networks (DNNs) training, and they usually train a DNN for each single visual recognition task, leading to a laborious and time-consuming visual recognition paradigm. To address the two challenges, Vision-Language Models (VLMs) have been intensively investigated recently, which learns rich vision-language correlation from web-scale image-text pairs that are almost infinitely available on the Internet and enables zero-shot predictions on various visual recognition tasks with a single VLM. This paper provides a systematic review of visual language models for various visual recognition tasks, including: (1) the background that introduces the development of visual recognition paradigms; (2) the foundations of VLM that summarize the widely-adopted network architectures, pre-training objectives, and downstream tasks; (3) the widely-adopted datasets in VLM pre-training and evaluations; (4) the review and categorization of existing VLM pre-training methods, VLM transfer learning methods, and VLM knowledge distillation methods; (5) the benchmarking, analysis and discussion of the reviewed methods; (6) several research challenges and potential research directions that could be pursued in the future VLM studies for visual recognition. A project associated with this survey has been created at https://github.com/jingyi0000/VLM_survey.

1 INTRODUCTION

Visual recognition has progressed from task-specific DNN training toward VLM pre-training and zero-shot prediction using web-scale image-text pairs. This survey organizes the emerging VLM literature across methods, tasks, benchmarks, and future directions.

  • Motivation: Visual recognition supports applications including autonomous driving, remote sensing, and robotics but requires costly task-specific labelled data under conventional DNN training.Deep learning also introduces slow convergence when DNNs are trained from scratch.
  • Paradigm development: Pre-training followed by fine-tuning accelerates convergence and improves performance, yet still requires labelled fine-tuning data for every downstream task.The paradigm uses large-scale annotated or unannotated data before task-specific adaptation.
  • VLM paradigm: VLM pre-training learns image-text correspondences from abundant web image-text pairs and supports downstream zero-shot prediction without task-specific fine-tuning.This paradigm is guided by vision-language objectives and uses web data that are almost infinitely available on the Internet.
  • VLM paradigm: Pretrained CLIP achieved superior zero-shot performance across 36 visual recognition tasks spanning image classification, human action recognition, and optical character recognition.The result illustrates the breadth of downstream tasks supported by a single pretrained VLM.
  • Related research: Beyond pre-training, research investigates transfer learning approaches such as prompt tuning and visual adaptation, alongside knowledge distillation for downstream adaptation.These lines of work target effective adaptation or use of pretrained VLMs across downstream tasks.
  • Survey scope: The survey reviews VLM visual recognition studies across backgrounds, foundations, datasets, technical approaches, benchmarking, and future research directions.It covers image classification, object detection, semantic segmentation, pre-training, transfer learning, and knowledge distillation.

2 BACKGROUND

Visual recognition evolved from hand-crafted features and task-specific training toward pre-training and fine-tuning, then to VLM pre-training with zero-shot prediction. This survey reviews that progression and organizes VLM research around objectives, frameworks, and related methods.

  • Visual recognition progressed through five paradigms, from traditional machine learning to VLM pre-training and zero-shot prediction.
  • Traditional machine learning depended on expert-designed features and lightweight models, limiting performance on complex tasks and scalability.
  • Deep neural networks replaced feature engineering with end-to-end learning, while supervised pre-training transferred knowledge from large labelled datasets to downstream tasks.
  • Unsupervised pre-training reduced dependence on labelled pre-training data, but supervised fine-tuning with labelled downstream data remained necessary.
  • VLM pre-training learns image-text correspondences from large-scale web image-text pairs and supports downstream zero-shot prediction without task-specific fine-tuning.
  • The survey presents VLM research through pre-training objectives, frameworks, transfer learning, knowledge distillation, and the scope distinction from surveys of vision-language tasks.

3 VLM FOUNDATIONS

VLM foundations combine image and text encoders with vision-language objectives that learn cross-modal representations for zero-shot and other downstream evaluations. The survey organizes architectures, objectives, pre-training frameworks, and evaluation setups across visual recognition tasks.

  • Core components: VLMs encode paired images and texts into embeddings, then match those embeddings to support zero-shot evaluation on unseen data.The approach uses an image encoder and a text encoder trained with vision-language objectives.
  • Network architectures: CNN-based and Transformer-based architectures are widely used to learn image features, while Transformers are widely adopted for text features.Examples of image architectures include VGG, ResNet, and EfficientNet.
  • Pre-training objectives: Vision-language pre-training objectives fall into contrastive, generative, and alignment categories.These objectives are designed to learn rich vision-language correlation.
  • Pre-training objectives: Contrastive objectives pull paired samples together and push non-paired samples apart in feature space.Image-text contrastive learning commonly uses a symmetric InfoNCE loss contrasting images with text keys and texts with image keys.
  • Pre-training objectives: Generative objectives learn semantic features by generating image, text, or cross-modal data, including masked reconstruction and image-to-text generation.Masked image modelling reconstructs masked patches conditioned on unmasked patches.
  • Frameworks and evaluation: Alignment objectives model global image-text or local region-word correspondence, with local matching supporting dense tasks such as object detection.The survey identifies two-tower, two-leg, and one-tower frameworks, and evaluates zero-shot prediction, linear probing, and several downstream tasks.

4 DATASETS

VLM pre-training relies on large-scale internet image-text datasets, while evaluation uses datasets spanning multiple visual recognition and multimodal tasks. These resources are larger and cheaper to collect than traditional crowd-labelled datasets.

  • Pre-training datasets: Internet-sourced image-text datasets are widely used for VLM pre-training and are generally collected at billion scale.The survey contrasts these resources with traditional crowd-labelled datasets.
  • Pre-training datasets: Image-text datasets are much larger and cheaper to collect than traditional crowd-labelled datasets.Several VLM studies also use auxiliary datasets to provide additional information for vision-language learning.
  • Evaluation datasets: VLM evaluations include 27 image classification datasets, 4 object detection datasets, 4 semantic segmentation datasets, 2 image-text retrieval datasets, and 3 action recognition datasets.The image classification collection ranges from fine-grained recognition to general visual recognition tasks.

5 VISION-LANGUAGE MODEL PRE-TRAINING

VLM pre-training uses contrastive, generative, and alignment objectives to learn vision-language representations from image-text data. The survey categorizes these objectives, datasets, and methods while noting complementary strengths and limitations.

  • Pre-training objectives: VLM pre-training objectives fall into contrastive, generative, and alignment categories.The survey reviews these objectives across multiple VLM pre-training studies.
  • Contrastive objectives: Contrastive learning pulls paired image-text embeddings together and pushes non-matching pairs apart, enabling zero-shot downstream predictions.CLIP uses a symmetrical image-text InfoNCE loss based on image-text embedding similarity.
  • Contrastive objectives: Scaling pre-training to 1.8 billion noisy image-text pairs improves robustness, while other methods reduce data requirements through pseudo-pairing, sampling, augmentation, or neighbor supervision.ALIGN exemplifies large-scale noisy-data training; DeCLIP, OTTER, and ZeroVL target data efficiency.
  • Discussion: Contrastive objectives face challenges from jointly optimizing positive and negative pairs and selecting a heuristic temperature hyper-parameter.These limitations complicate optimization and control feature discriminability.
  • Generative objectives: Generative objectives mask or reconstruct image, language, or cross-modal content and can generate captions to capture contextual and fine-grained vision-language knowledge.Examples include masked image modelling, masked language modelling, masked cross-modal modelling, and image-to-text generation.
  • Alignment objectives: Alignment objectives are easy to optimize and support local image-text matching, but often learn limited within-modality correlations, so they are commonly auxiliary losses.Their role is to enhance cross-modal correlation alongside other pre-training objectives.

6 VLM TRANSFER LEARNING

VLM transfer learning adapts pre-trained models to downstream distribution and objective gaps through prompt tuning, feature adapters, and other modifications. Prompt tuning is parameter-efficient and black-box compatible, whereas feature adaptation is more flexible but changes the architecture.

  • Motivation: VLM transfer addresses gaps between pre-training and downstream image-text distributions and between task-agnostic and task-specific objectives.Downstream tasks may require coarse- or fine-grained classification and region- or pixel-level recognition.
  • Transfer setups: Transfer methods include prompt tuning, feature adapters, and other approaches across supervised, few-shot supervised, and unsupervised setups.These setups differ in how much labelled or unlabelled downstream data is used.
  • Prompt tuning: Prompt tuning learns text, visual, or joint text-visual prompts while keeping the original VLM largely unchanged.Text prompt methods learn context vectors, while visual methods modulate image-encoder inputs; joint methods optimize both modalities.
  • Discussion: Prompt tuning introduces few parameters and supports efficient black-box transfer, but its prompts remain constrained by the original VLM manifold.This limits flexibility despite avoiding extensive network modification.
  • Feature adaptation: Feature adaptation inserts lightweight trainable adapters after image and language encoders while freezing the original VLM parameters.This approach can tailor features to varied downstream tasks, including complex ones.
  • Discussion: Prompt tuning and feature adapters both enable efficient transfer with limited additional parameters, while unsupervised transfer can achieve competitive performance on various tasks.The original VLM is frozen in these parameter-efficient approaches.

7 VLM KNOWLEDGE DISTILLATION

VLM knowledge distillation transfers broad image-level knowledge into task-specific models for dense recognition, especially object detection and semantic segmentation. The survey organizes methods around vocabulary expansion and image-to-region or image-to-pixel representation alignment.

  • Motivation: Knowledge distillation transfers general VLM knowledge to task-specific dense-prediction models without retaining the original VLM architecture.This allows task-specific designs for object detection and semantic segmentation.
  • Distillation targets: Most methods address the mismatch between image-level VLM representations and region- or pixel-level requirements in detection and segmentation.The survey identifies these two dense visual recognition tasks as the main distillation targets.
  • Object detection: Open-vocabulary object detection distills VLM knowledge to detect objects described by arbitrary texts beyond base categories.ViLD aligns detector embeddings with the CLIP image-encoder embedding space.
  • Object detection: Detection distillation also uses continuous or regional prompt learning and VLM-predicted pseudo-labels to improve open-vocabulary detectors.DetPro learns detection prompts, PromptDet aligns regional embeddings, and other methods use pseudo-labels.
  • Semantic segmentation: Open-vocabulary semantic segmentation combines class-agnostic segment grouping with VLM recognition or uses lightweight decoders, mask proposals, and pixel-level pseudo-labels.These methods target pixels described by arbitrary texts beyond base classes.
  • Methodology: Distillation methods include feature-space consistency, prompt learning, and pseudo-label strategies for transferring VLM knowledge to dense tasks.The survey frames these strategies around aligning image-level knowledge with object- or pixel-level representations.

8 PERFORMANCE COMPARISON

The survey compares VLM pre-training, transfer learning, and knowledge distillation across visual recognition tasks, finding strong image-classification zero-shot performance but weaker dense-recognition development and notable benchmarking constraints.

  • VLM pre-training: VLMs achieve strong zero-shot image-classification performance, with larger training datasets and model sizes consistently improving results.COCA achieves state-of-the-art performance on ImageNet, while FILIP performs consistently across 11 tasks.
  • VLM pre-training: VLMs also enable effective zero-shot prediction for object detection and semantic segmentation through local vision-language objectives such as region-word matching.The survey notes that these dense-prediction results may not align with conclusions from image-classification evaluations.
  • VLM transfer learning: VLM transfer consistently improves downstream image classification by mitigating domain gaps through labelled or unlabelled task-specific data.On ImageNet, Wise-FT, CoOp, and TPT improve accuracy by 10.9%, 1.7%, and 0.8%, respectively.
  • VLM transfer learning: Few-shot supervised transfer trails supervised transfer, while unsupervised transfer can match or exceed few-shot methods but remains vulnerable to noisy pseudo labels.WiseFT reaches 87.1% versus CuPL at 76.6%; UPL outperforms 2-shot CoOp by 0.4%.
  • VLM knowledge distillation: VLM knowledge distillation consistently improves object detection and semantic segmentation by adding general VLM knowledge to task-specific model designs.The reviewed evaluations cover detection datasets including COCO and LVIS and segmentation datasets including PASCAL VOC and ADE20k.
  • Summary: Dense visual recognition research lags behind image classification, while unsupervised transfer remains promising but comparatively neglected.Supervised and few-shot transfer still require labelled images.
  • Benchmarking: Fair benchmarking is difficult for VLM pre-training and knowledge distillation because studies vary in data, networks, task-specific backbones, and computational requirements.The survey reports that these areas lack norms for training data, networks, and downstream tasks.

9 FUTURE DIRECTIONS

The survey identifies future directions spanning more efficient and capable VLM pre-training, broader transfer-learning settings, and expanded knowledge distillation across models and tasks.

  • VLM pre-training: Fine-grained vision-language correlation modeling could improve patch- and pixel-level recognition for dense prediction tasks.The survey describes local correspondence knowledge as beneficial for object detection and semantic segmentation.
  • VLM pre-training: Unifying vision and language within a single Transformer may improve cross-modal communication, training effectiveness, and training efficiency.This direction contrasts with existing VLMs that use separate networks for the two modalities.
  • VLM pre-training: Multilingual VLM pre-training could reduce cultural and regional bias and support visual recognition across different language scenarios.The survey notes that most existing VLMs are trained primarily with English text.
  • VLM pre-training: Data-efficient VLMs could reduce sustainability concerns by learning effectively from limited image-text data and supervision across image-text pairs.Current approaches generally rely on large-scale data and intensive computation.
  • VLM pre-training: Using LLMs to augment raw image-text pairs is proposed to provide richer language knowledge for learning vision-language correlations.The survey expects further exploration of LLMs in VLM pre-training.
  • VLM transfer learning: Unsupervised, visual-prompt, test-time, and LLM-assisted transfer are proposed to reduce labelled-data needs, repetitive training, or adaptation limitations.These directions target unlabelled data, dense prediction, on-the-fly prompt adaptation, and automatic prompt generation.
  • VLM knowledge distillation: Knowledge distillation could combine multiple VLMs and extend beyond detection and semantic segmentation to additional recognition tasks.Suggested tasks include instance segmentation, panoptic segmentation, and person re-identification.

10 CONCLUSION

The survey synthesizes VLM datasets, methods, benchmarks, and future directions for visual recognition, emphasizing web-data usage and zero-shot prediction without task-specific fine-tuning.

  • Conclusion: VLMs use web data to support zero-shot visual recognition without task-specific fine-tuning across a wide range of tasks.The survey characterizes this paradigm as simple to implement and successful in visual recognition.
  • Conclusion: Comparative tables of datasets, approaches, and performance provide a broad view of recent VLM pre-training development.The survey presents this synthesis as support for future research in the field.

B. DATASETS FOR PRE-TRAINING VLM

VLM pre-training uses internet-collected image-text datasets that are substantially larger and cheaper to collect than traditional crowd-labelled datasets.

  • Pre-training datasets: Internet-collected image-text datasets for VLM pre-training are much larger and cheaper to collect than traditional crowd-labelled datasets.Recent image-text datasets are generally at billion scale.

B.1. Image-Text Datasets

Image-text datasets for VLM research range from millions to billions of web-collected pairs, with increasing multilingual coverage and varied annotation quality. Evaluation resources additionally include captioning, grounding, detection, and broad visual recognition annotations.

  • Structured annotations: Specialized resources provide richer supervision through multiple captions, scene graphs, visual question answers, local word regions, and object-phrase grounding.COCO Caption offers 5 or 40 reference captions; VG supplies object-level information, scene graphs, and visual question-answer pairs; LR localizes every word; Gold-G provides 0.8M grounding examples.
  • Data quality and filtering: Dataset scale is often accompanied by noisy or weakly filtered data, as illustrated by LAION400M’s CLIP filtering, CC12M’s relaxed collection pipeline, and ALIGN’s noisy pairs.CC12M explicitly trades precision for a larger 12M-pair collection, while ALIGN contains 1.8B noisy pairs.
  • Web-scale pretraining datasets: Web-scale pretraining datasets span 1M to over 5.8B image-text pairs collected from websites, social media, and other public internet sources.Examples include SBU with 1M pairs, Red Caps with 12M, LAION400M with 400M, and LAION5B with over 5.8B.
  • Multilingual coverage: Multilingual resources extend VLM pretraining across 108–109 languages, including WIT, WebLI, and the Chinese WuKong dataset.WIT contains 37.6M pairs across 108 languages, WebLI contains 10B images and 12B texts across 109 languages, and WuKong contains 100M Chinese pairs.
  • Evaluation coverage: VLM evaluation draws on 27 image classification, 4 object detection, 4 semantic segmentation, 2 image-text retrieval, and 3 action recognition datasets.The survey organizes these evaluation datasets across recognition, localization, retrieval, and video-action tasks.

C.1. Datasets for Image Classification

Image-classification evaluations cover fine-grained objects, scenes, textures, emotions, geolocation, remote sensing, medical imagery, text-containing images, and video actions. Their datasets vary substantially in scale, modality, labels, and supervision.

  • Fine-grained recognition: The evaluation suite spans fine-grained recognition of food, pets, flowers, birds, cars, and aircraft across diverse class counts and annotation formats.Datasets include class labels alongside pixel masks, head or object boxes, part coordinates, attributes, and hierarchical categories.
  • General recognition benchmarks: Standard classification benchmarks range from CIFAR-10’s 60,000 ten-class images to ImageNet-1k’s approximately 1.2M samples across 1,000 categories.CIFAR-100 also contains 60,000 samples, organized into 100 categories and 20 super-categories.
  • Reasoning and supervision: Several datasets test visual reasoning or recognition under limited-label settings, including CLEVR Counts, STL-10, and Oxford-IIIT PETS with multiple annotation types.CLEVR Counts evaluates counting, STL-10 separates unsupervised and supervised training subsets, and Oxford-IIIT PETS provides breed, pixel-wise body, and head-box annotations.
  • Action recognition: Video action recognition is evaluated with UCF101 and Kinetics700, covering 101 and 700 actions respectively.UCF101 contains about 13K YouTube clips, while Kinetics700 contains about 65K clips, each lasting around 10 seconds.
Loading 2304.00685v2…