PaperlayerView exact revision on arXiv

Source-linked AI summary

An Introduction to Vision-Language Modeling

Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C. Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, Mark Ibrahim, Melissa Hall, Yunyang Xiong, Jonathan Lebensold, Candace Ross, Srihari Jayakumar, Chuan Guo, Diane Bouchacourt, Haider Al-Tahan, Karthik Padthe, Vasu Sharma, Hu Xu, Xiaoqing Ellen Tan, Megan Richards, Samuel Lavoie, Pietro Astolfi, Reyhane Askari Hemmat, Jun Chen, Kushal Tirumala, Rim Assouel, Mazda Moayeri, Arjang Talattof, Kamalika Chaudhuri, Zechun Liu, Xilun Chen, Quentin Garrido, Karen Ullrich, Aishwarya Agrawal, Kate Saenko, Asli Celikyilmaz, Vikas Chandra

arXiv:2405.17247v1cs.LG

TL;DR

VLM research seeks reliable ways to connect high-dimensional visual inputs with discrete language, despite challenges in reasoning, bias, hallucination, and temporal understanding. This paper introduces what VLMs are, how they are trained, and how they can be evaluated, including video extensions. It emphasizes that progress depends on data quality, grounding, human alignment, and improved benchmarks, while some approaches remain computationally impractical.

  • Problem

    Connecting vision to language remains difficult because VLMs can struggle with spatial relationships, counting, benchmark bias, hallucinations, and temporal understanding.

  • Method

    The paper provides an accessible introduction to VLM definitions, training approaches such as CLIP, and evaluation methods for visual-linguistic abilities, biases, hallucinations, and videos.

  • Results

    The paper synthesizes VLM training and evaluation practices, including findings that some video models perform at random on physics reasoning while humans exceed 80% accuracy.

  • Takeaways & Limitations

    Reliable VLM progress requires high-quality image-caption data, stronger grounding and human alignment, and careful attention to benchmark limitations.

  • Takeaways & Limitations

    Conditional diffusion-model classification can require hundreds or thousands of network evaluations per test image, making inference impractical without further optimization.

Abstract

from arXiv · show

Following the recent popularity of Large Language Models (LLMs), several attempts have been made to extend them to the visual domain. From having a visual assistant that could guide us through unfamiliar environments to generative models that produce images using only a high-level text description, the vision-language model (VLM) applications will significantly impact our relationship with technology. However, there are many challenges that need to be addressed to improve the reliability of those models. While language is discrete, vision evolves in a much higher dimensional space in which concepts cannot always be easily discretized. To better understand the mechanics behind mapping vision to language, we present this introduction to VLMs which we hope will help anyone who would like to enter the field. First, we introduce what VLMs are, how they work, and how to train them. Then, we present and discuss approaches to evaluate VLMs. Although this work primarily focuses on mapping images to language, we also discuss extending VLMs to videos.

1 Introduction

VLM research extends language models with visual inputs but still faces reliability challenges, including spatial understanding and counting. This introduction explains VLMs, training recipes, evaluation methods, and video extensions for newcomers to the field.

  • Motivation: VLMs extend language models with visual inputs to support applications connecting vision and language.The paper motivates this connection as important for current AI applications.
  • Motivation: Reliable vision-language understanding remains difficult because models struggle with spatial relationships and counting without additional engineering and annotation.
  • Scope: The paper provides an accessible introduction rather than a complete survey, covering what VLMs are, how they are trained, and how they are evaluated.It is intended especially for students and researchers entering the field.
  • Training: Training coverage includes contrastive, masking, generative, and pretrained-backbone paradigms, together with dataset, curation, encoder, loss, grounding, and alignment choices.
  • Evaluation: Evaluation coverage discusses vision-language benchmarks, their strengths and weaknesses, and methods for measuring bias.
  • Video and broader motivation: The introduction also addresses video-language modeling, where higher computation and temporal mapping create challenges beyond image-based VLMs.

2 The Families of VLMs

The paper categorizes transformer-based VLM initiatives into contrasting, masking, generative, and pretrained-backbone families. These categories overlap, but they help organize the field and clarify how models connect visual and linguistic representations.

  • Families of VLMs: Recent transformer-based VLMs are organized into four training paradigms: contrastive, masking, generative, and pretrained-backbone approaches.
  • Overlapping paradigms: The families are not mutually exclusive because many models combine contrastive, masking, and generative criteria.
  • Pretrained backbones: Pretrained-backbone VLMs learn a mapping between an image encoder and an existing language model, often reducing computational cost relative to training both encoders from scratch.
  • Generative models: Generative VLMs produce images or captions, whereas other approaches commonly use intermediate representations or partial reconstructions.

2.1 Early work on VLMs based on transformers

Early transformer-based VLMs adapted masked-language-modeling ideas to multimodal inputs by combining text and image tokens. Visual-BERT and ViLBERT exemplify this approach.

  • Early transformer VLMs: Visual-BERT and ViLBERT extend BERT-style language modeling to visual data by combining text tokens with image tokens.
  • Training objectives: These models are trained with objectives that include a classical masked-modeling task for reconstructing masked content.

2.2 Contrastive-based VLMs

Contrastive VLMs learn aligned representations by distinguishing positive from negative examples, using objectives such as NCE and InfoNCE. CLIP applies this principle to image-caption pairs, while later variants modify the loss or account for caption diversity.

  • Energy-based view: Energy-based contrastive training assigns low energy to observed data and high energy to unobserved data.
  • Noise-contrastive estimation: Noise-contrastive estimation replaces intractable model-distribution negatives with samples from a noise distribution and trains the model to discriminate real from noisy examples.
  • InfoNCE: InfoNCE selects the positive pair through a softmax over similarity scores while assigning lower probability to negative pairs.The similarity can be computed with metrics such as cosine similarity in representation space.
  • CLIP: CLIP trains vision and text encoders in a shared representation space using image-caption positives and other batch captions as negatives.Its original training used 400 million caption-image pairs collected from the web.
  • CLIP: 76.2% zero-shot classification accuracy was achieved by ResNet-101 CLIP, matching a supervised ResNet and surpassing it on several robustness benchmarks.
  • Variants: SigLIP replaces CLIP’s InfoNCE objective with binary-cross-entropy NCE, enabling better zero-shot performance at smaller batch sizes.
  • Variants: Llip conditions image encoding on the target caption through cross-attention, generally improving zero-shot transfer classification and retrieval performance.

2.3 VLMs with masking objectives

Masking-based VLMs learn multimodal representations by reconstructing masked image or text content, while information-theoretic analyses frame their objectives as balancing retained predictive information against discarded information.

  • Masking-based approach: Masking treats structured corruption as denoising, extending masked-token prediction and inpainting ideas to multimodal representation learning.The approach uses spatial masking for images and token masking for text.
  • Masking-based approach: FLAVA exemplifies masking-based VLMs with transformer components specialized for image and text processing.Its image encoder converts image patches into transformer representations, while its text encoder produces contextual hidden states.
  • Masking-based approach: MaskVLM reduces reliance on pretrained vision encoders by masking directly in pixel and text-token spaces and exchanging information across modalities.Text reconstruction receives image-encoder information, and image reconstruction receives text-encoder information.
  • Information-theoretic view: An information-theoretic view interprets VLM learning as a rate-distortion problem that removes superfluous information while preserving predictive information.The rate term acts as an entropy bottleneck, whereas the distortion term is realized through auto-encoding.
  • Information-theoretic view: Contrastive and auto-encoding losses implement distortion, while the data transformation largely determines the information rate.

2.4 Generative-based VLMs

Generative VLMs model the generation of text and/or images, using tokenized multimodal sequences or conditional image likelihoods to support both generation and downstream understanding tasks. Their generative classifiers offer robustness and compositional advantages, but inference can remain computationally impractical.

  • Generative paradigm: Generative VLMs differ from latent-representation approaches by directly generating text, images, or both.Examples include captioning systems and mixed-modal models that generate interleaved textual and image content.
  • Generative paradigm: CoCa combines contrastive and generative losses, enabling multimodal understanding tasks such as VQA without additional multimodal fusion adaptation.
  • Autoregressive generation: CM3Leon tokenizes images and text into interleaved sequences processed by a decoder-only transformer, using a special token for modality transitions.Its image tokenizer encodes a 256 × 256 image into 1024 tokens from an 8192-token vocabulary.
  • Autoregressive generation: CM3Leon uses retrieval-augmented pretraining, prepending relevant multimodal documents before next-token prediction to increase data efficiency.
  • Downstream tasks: Generative image models can perform classification or caption prediction without retraining by estimating conditional image likelihoods and applying Bayes’ theorem.
  • Limitations: Conditional diffusion classification remains computationally expensive because it scales with the number of classes and may require hundreds or thousands of network evaluations per test image.The passage states that inference is impractical until further optimizations are developed.
  • Downstream tasks: Generative classifiers improve out-of-distribution robustness and compositional reasoning relative to discriminative methods, while also aligning better with human judgments.They can additionally be adapted with unlabeled test examples for classification, segmentation, and depth prediction under distribution shift.

2.5 VLMs from Pretrained Backbones

VLMs built from pretrained visual and language backbones learn mappings between existing representations rather than training every component from scratch. This reduces computational demands and supports rapid adaptation, while model designs increasingly unify diverse vision-language tasks.

  • Motivation: Training VLMs from scratch can require hundreds to thousands of GPUs and hundreds of millions of image-text pairs.Leveraging existing LLMs and visual extractors is motivated by the lower compute required to learn their mapping.
  • Frozen: Frozen connects a vision encoder to a frozen 7-billion-parameter language model through a lightweight mapping network trained for text generation.The language model remains frozen to preserve features learned during pretraining.
  • Frozen: Frozen supports rapid adaptation, general-knowledge access, and visual-linguistic binding, although its performance is described as modest.
  • MiniGPT family: MiniGPT-4 aligns image representations with an LLM through a simple linear projection, while MiniGPT-5 adds interleaved image outputs using generative visual tokens.MiniGPT-5 feeds mapped visual features into a frozen Stable Diffusion 2.1 model.
  • Unified interfaces: MiniGPT-v2 uses task-specific identifiers to unify image captioning, visual question answering, and object grounding through one interface.Experiments report strong vision-language understanding on visual question answering and visual grounding benchmarks.
  • Shared principles: Qwen-VL and BLIP-2 similarly combine pretrained LLMs, visual encoders, and alignment modules that compress or map visual representations into the LLM input space.BLIP-2 trains a relatively small Q-Former while keeping its major pretrained components frozen.
  • Shared principles: The introduction presents selected pretrained-backbone models because they share the principle of learning mappings between representations rather than covering the full literature.

3 A Guide to VLM Training

Effective VLM training depends on carefully curated, diverse multimodal data, improved captions, grounding, and alignment. The section also covers contrastive and self-supervised augmentation strategies, interleaved datasets, and methods for assessing data quality.

  • Training data: Diverse and balanced datasets support broad concept coverage, while deduplication reduces compute and memorization risks.
  • Training data: Data pruning uses heuristics, pretrained-VLM alignment scores, or sampling strategies designed to improve dataset diversity and balance.
  • Training data: Perfectly balanced web datasets are impractical, and downstream performance remains uneven because zero-shot abilities depend strongly on concept coverage in training data.
  • Training data: Caption quality can be improved through filtering and synthetic generation, with consistent and complete captions linked to better model training outcomes.
  • Using data augmentation: Self-supervised augmentation adds a regularization signal, while CLIP-rocket extends it cross-modally using weak and strong image-text augmentations with separate projectors.
  • Interleaved data: Interleaved datasets pair text and images through contextual relevance, enabling existing text corpora to support multimodal learning despite possible context limitations.
  • Assessing multimodal data quality: Multimodal data quality spans text, image, and image-text alignment, motivating scalable methods that identify high-quality training subsets.

3.2 Software

VLM development relies on software libraries for implementation and comparison, but training resources can be prohibitively expensive. Efficiency improvements in compilation, attention, data loading, and masking can reduce these costs, while design choices strongly affect performance.

  • Software: OpenCLIP and transformers implement many VLMs and provide platforms for benchmarking or comparing pretrained models on downstream tasks.
  • Compute resources: CLIP and OpenCLIP training has used more than 500 GPUs, corresponding to cloud costs of hundreds of thousands of dollars.
  • Efficiency: torch.compile and xformers can speed training, while data loading is another often-overlooked efficiency bottleneck.
  • Efficiency: Randomly masking image tokens can significantly accelerate training while improving model performance.
  • Design choices: Image resolution, visual encoder capacity, and visual pretraining data most affect performance, whereas the modality-connection choice is less important.
  • Design choices: The right mixture of text-only, interleaved, and image-caption data achieves the best performance across zero-shot classification and visual-question answering.

3.3 Which model to use?

VLMs can be built with contrastive, masking, generative, or pretrained-backbone approaches, each offering different capabilities and costs. Model selection must also account for grounding challenges, including relations, negation, counting, and attributes.

  • Model families: VLM training families include contrastive objectives, masking-based reconstruction, generative paradigms, and mappings between pretrained vision and language backbones.
  • Contrastive models: CLIP aligns image and text representations in a shared space, supporting text-to-image retrieval and serving as a foundation for richer grounding models.
  • Contrastive models: Unlike generative models, CLIP retrieves matching captions from an existing list rather than generating a caption for an image.
  • Masking models: Masking jointly models image and text distributions through reconstruction, but decoder training can add a bottleneck compared with contrastive learning.
  • Generative models: Generative VLMs use diffusion or autoregressive criteria to generate images from text, though the value of reconstruction for world modeling remains debated.
  • Pretrained backbones: Pretrained backbones reduce computational requirements by learning only a cross-modal mapping, but inherited hallucinations and biases may require correction.
  • Improving grounding: Grounding methods address missed prompt elements and hallucinations involving spatial relations, negation, counting, and visual attributes.
  • Improving grounding: Bounding-box supervision helps models associate textual descriptions with the corresponding visual regions and improve grounding.

3.5 Improving alignment

This section describes instruction tuning and RLHF as approaches for improving multimodal alignment and chat capabilities, then surveys LLaVA-family and related methods for handling visual and textual challenges.

  • Alignment methods: Instruction tuning fine-tunes VLMs on supervised instruction, input, and desired-response data, while RLHF uses a reward model trained on human preferences.Instruction-tuning datasets are typically much smaller than pretraining datasets; RLHF avoids requiring comparable supervised preference examples by using an auxiliary reward model.
  • A LLaVA story: LLaVA combines pretrained Vicuna and CLIP encoders through a linear projector and uses 150k synthetic visual instruction samples.The resulting model improved qualitative chat interactions and performance on synthetic instruction-following and Science QA benchmarks.
  • A LLaVA story: LLaVA 1.5 improves training efficiency with a cross-modal MLP and academic VQA data, using 600k image-text pairs and approximately one day on 8 A100 GPUs.It performs well on academic VQA and instruction-following benchmarks.
  • A LLaVA story: LLaVA-RLHF adapts RLHF with factual image information to reduce reward hacking and improve multimodal alignment.It reaches 94% of GPT-4’s performance level on LLaVA-Bench and outperforms baselines by 60% on the hallucination-focused MMHAL-BENCH.
  • A LLaVA story: LLaVA-NeXT increases image resolution, improves instruction-data mixtures, and scales its largest language backbone to 34B parameters.It achieves state-of-the-art performance among open-source multimodal LLMs and closes the gap with Gemini Pro.
  • Multimodal in-context learning: Otter demonstrates multimodal in-context learning, enabled by fine-tuning OpenFlamingo on MIMIC-IT’s roughly 2.8M multimodal instruction-response pairs.The dataset includes relevant instruction-image-answer examples paired with a test example, and Otter exhibits stronger instruction following.
  • Improving text-rich image understanding: Visual text understanding remains difficult for complex relationships, while text-rich instruction data and high-resolution patch processing target these limitations.LLaVAR uses OCR-derived data from 422K text-rich images, and Monkey processes images in uniform sliding-window patches.

3.7 Parameter-Efficient Fine-Tuning

This section presents parameter-efficient alternatives to full VLM fine-tuning, including LoRA, prompting, adapters, and mappings between frozen pretrained modules.

  • Parameter-Efficient Fine-Tuning: Parameter-efficient fine-tuning trains only a subset of parameters to reduce the computational cost of adapting large pretrained VLMs.The motivation is that fine-tuning the entire parameter set becomes impractical as pretrained models grow.
  • LoRA-based methods: LoRA adapts pretrained models with low-rank updates, while QLoRA combines it with a frozen 4-bit quantized backbone and DoRA separates weight magnitude from direction.These variants extend parameter-efficient adaptation across language and vision-language settings.
  • Prompt-based methods: Prompt-based methods adapt pretrained vision-language models through learned prompts rather than manual prompt design.The approach builds on shared image-text representations that support zero-shot transfer to downstream tasks.
  • Adapter-based methods: Adapter methods add bottleneck modules between pretrained layers and can blend adapted features residually with original visual or language features.Weight sharing with adapters rivals full fine-tuning while updating only 4.18% of parameters for image-text tasks and 3.39% for video-text tasks.
  • Mapping-based methods: Mapping-based methods train a lightweight mapping between frozen vision encoders and LLMs, avoiding adapter layers and reducing trainable parameters.LiMBeR projects each visual feature vector into the LLM hidden-state dimension.

4 Approaches for Responsible VLM Evaluation

Responsible VLM evaluation requires measuring image–language mapping across varied tasks while accounting for benchmark and metric limitations. The paper discusses evaluation methods spanning captioning, VQA, text understanding, classification, reasoning, and robustness.

  • Visio-linguistic benchmarking: Visio-linguistic benchmarks test whether models associate words or phrases with visual clues, from simple object detection to spatial reasoning.They include tasks ranging from questions such as whether a dog is visible to questions involving counts and scene relationships.
  • Caption evaluation: Captioning metrics such as BLEU, ROUGE, and exact string matching are heuristic or brittle proxies that may underestimate or misrepresent model performance.CLIPScore avoids reference-caption comparison but remains limited by the underlying CLIP model’s performance.
  • Visual Question Answering: VQA answers natural-language questions about images and can cover multiple-choice, open-ended, text-centric, and selective-prediction settings.Selective prediction evaluates whether models abstain from questions they would otherwise answer incorrectly while preserving accuracy on answered questions.
  • Text-centric evaluation: Text-centric evaluation extends VLM assessment to recognizing and reasoning over scene text, documents, extracted key information, and handwritten mathematical expressions.The paper lists separate benchmark families for text recognition, scene-text VQA, document VQA, key information extraction, and handwritten mathematical expression recognition.
  • Zero-shot classification: Zero-shot classification results depend strongly on prompt engineering and whether evaluated concepts appeared in training data, complicating the zero-shot label.Automatically generated prompts, rich visual descriptions, and synonym substitution can substantially change performance.
  • Benchmark reliability: Compositional and robustness benchmarks expose evaluation failures caused by nonsensical negatives, equal-probability ties, and unimodal shortcuts in benchmark data.Balanced benchmarks can penalize language-only shortcuts, while common VQA question patterns may make answers predictable without using the image.

5 Extending VLMs to Videos

Video-language models extend image-language approaches to temporal data, enabling motion-aware understanding and new evaluations, but they face substantial data, compute, and reasoning challenges.

  • Video-language modeling introduces capabilities beyond images, including understanding object motion, dynamics, and spatiotemporal localization.
  • VideoBERT established an early successful general video-language approach using early fusion of visual and textual tokens.
  • VideoOFA adapts a generative image-text VLM to video through intermediate pretraining for captioning, matching, and frame-order modeling.
  • VideoOFA showed improved performance over previous models on video captioning and question-answering benchmarks, including MSVD, MSRVTT, TGIF, and TVQA.
  • Synthetic physics videos reveal weak reasoning: VideoLLaMA and PandaGPT perform no better than random, while humans exceed 80% accuracy.
  • Video-text pretraining is limited by scarce temporal supervision and high computational cost, causing scene-focused data to encourage image-like rather than motion-aware modeling.

6 Conclusion

VLM research remains active across contrastive and generative methods, with progress depending on data quality, grounding, alignment, and careful evaluation across image and video settings.

  • High compute and data costs motivate leveraging pretrained LLMs or image encoders to learn mappings between modalities.
  • Large-scale high-quality images and captions, improved grounding, and alignment with human preferences are identified as important for reliable performance.
  • VLM benchmarks measure vision-linguistic and reasoning abilities, but some can be solved using language priors rather than visual evidence.
  • Video is an important modality for learning representations, yet substantial challenges remain before good video representations can be learned.

Acronyms

The paper uses a broad set of acronyms for vision-language models, learning methods, architectures, datasets, and evaluation concepts.

  • Architectures: BERT, ViT, CNN, and VQ-VAE expand to Bidirectional Encoder Representations from Transformers, Vision Transformer, Convolutional Neural Network, and Vector Quantised-Variational AutoEncoder.
  • Vision-language methods: CLIP means Contrastive Language–Image Pre-training; BLIP means Bootstrapping Language-Image Pre-training; CoCa means Contrastive Captioner.
  • Training and adaptation: SSL, MLM, MIM, NCE, EBM, MCMC, PEFT, LoRa, and RLHF denote learning, optimization, modeling, and adaptation techniques.
  • Model families: VLMs means Vision Language Models, while LLMs means Large Language Models and MLLMs means Multimodal Large Language Models.
  • Evaluation and resources: VQA, ARO, IoU, ROI, OOD, STR, FFCV, OBELICS, PUG, MERLOT, FLAVA, and DCI name evaluation concepts, data resources, systems, or benchmarks.
Loading 2405.17247v1…