An Introduction to Vision-Language Modeling

• VLMs enable important vision-language applications, but remain unreliable because they struggle with spatial relationships, counting, attributes, ordering, prompt adherence, and hallucination.These limitations make reliable VLM development an active research area.Source: §1 · p. 4
• The work provides an accessible introduction rather than a comprehensive survey, focusing on clear explanations and effective practices for students and researchers entering VLM research.It explicitly does not aim to cite every work or capture every best practice in the field.Source: §1 · p. 4
• It organizes VLMs into training families involving contrastive methods, masking or generative components, and pretrained backbones, while acknowledging overlap among categories.The categorization is intended to clarify inner mechanisms and help new researchers navigate the field.Source: §1 · pp. 4–5
• The introduction presents training recipes covering dataset selection, data curation, encoder reuse, contrastive versus generative objectives, performance improvement, grounding, and alignment.It frames these choices as dependent on different research goals and VLM capabilities.Source: §1 · p. 5
• It examines VLM benchmarks, their strengths and limitations, bias measurement, and video-language methods, including higher computational costs and temporal-mapping challenges.The evaluation discussion emphasizes robust and reliable assessment, while the video discussion identifies challenges beyond those of image understanding.Source: §1 · p. 5

2 The Families of VLMs

• Transformer-based VLMs are categorized into four training paradigms: contrastive, masking, pretrained-backbone mapping, and generative modeling.The paper presents one or two models per paradigm for high-level design insight.Source: §2 · pp. 5–6

Contrastive training

• Contrastive training aligns representations for positive image-text pairs while separating representations for negative pairs.It is described as a commonly used strategy based on positive and negative examples.Source: §2 · p. 5

Masking

• Masking trains VLMs to reconstruct masked image patches from unmasked text or masked caption words from an unmasked image.Source: §2 · p. 5

Pretrained backbones

• Pretrained-backbone approaches map an image encoder to an open-source LLM such as Llama, reducing computation relative to training both encoders from scratch.Source: §2 · p. 6

Generative VLMs

• Generative VLMs produce images or captions directly and are often the most expensive to train, while paradigms may be combined across approaches.Source: §2 · p. 6

Transformer-based architectures

• Visual-BERT [Li et al., 2019] and ViLBERT extended BERT to process visual data by combining text with image tokens.Source: §2.1 · p. 6

Training objectives

• The models were trained with masked modeling and sentence-image prediction objectives.The first predicts missing input content, while the second predicts whether a caption describes an image.Source: §2.1 · p. 6

Task performance

• Attention mechanisms helped the models associate words with visual clues, supporting strong performance across several vision-language tasks.Source: §2.1 · p. 6

2.2 Contrastive-based VLMs

• Contrastive training can be viewed as an energy-based model that assigns low energy to data from the target distribution and higher energy to other samples.The model uses an energy function Eθ(x), with a corresponding Boltzmann density, and maximum likelihood can in principle estimate the target distribution.Source: §2.2 · p. 6
• Because sampling negatives from the model distribution is often intractable, contrastive methods approximate it using MCMC, score matching, denoising score matching, or Noise Contrastive Estimation (NCE).NCE samples negatives from a noise distribution and formulates discrimination between real and noisy samples as binary classification with cross-entropy.Source: §2.2 · p. 8
• InfoNCE identifies the positive pair as most likely using representation-space similarity while assigning lower probability to negative pairs, but its effectiveness depends on mini-batch content and often requires large batches.In SimCLR, positives are an image and its augmented version, whereas negatives pair that image with other images in the mini-batch.Source: §2.2 · p. 8

2.2.1 CLIP

• 76.2% zero-shot classification accuracy enabled ResNet-101 CLIP to match supervised ResNet performance and surpass it on several robustness benchmarks after training on 400 million image-caption pairs.CLIP maps images and captions into a shared representation space using randomly initialized vision and text encoders, with captions from the same image as positives and other batch captions as negatives.Source: §2.2.1 · p. 9
• SigLIP [Zhai et al., 2023b] replaces CLIP’s InfoNCE objective with binary-cross-entropy NCE for better zero-shot performance at smaller batch sizes, while Llip [Lavoie et al., 2024] conditions image encoding on captions to improve zero-shot transfer classification and retrieval.Llip models caption diversity through a cross-attention module, increasing representation expressivity.Source: §2.2.1 · p. 9

2.3 VLMs with masking objectives

• Masking extends denoising and masked language or image modeling to VLMs, including FLAVA, MaskVLM, and information-theoretic connections.Masking is suited to transformers because tokenized inputs enable random token removal; related approaches include denoising autoencoders, inpainting, MLM, and MIM.Source: §2.3 · p. 9

2.3.1 FLAVA

• FLAVA combines image, text, and multimodal transformer encoders with masked modeling and contrastive losses, achieving state-of-the-art performance across 35 vision, language, and multimodal tasks.It is pretrained on 70 million publicly available image-text pairs; the multimodal encoder fuses modality-specific hidden states through projections and cross-attention.Source: §2.3.1 · pp. 9–10

2.3.2 MaskVLM

• MaskVLM reduces reliance on third-party vision encoders by masking directly in pixel and text-token spaces and transferring information between modalities during reconstruction.Text reconstruction receives image-encoder information, while image reconstruction receives text-encoder information.Source: §2.3.2 · p. 10

2.3.3 Information theoretic view on VLM objectives

• The information-theoretic view frames masking and other transformations as rate-distortion problems that remove superfluous information while preserving predictive information.The rate is bounded by an entropy bottleneck and largely determined by the transformation, whereas auto-encoding and contrastive losses implement distortion through reconstruction or representation equivalence.Source: §2.3.3 · pp. 10–11

2.4 Generative-based VLMs

• Generative-based VLMs include captioning models such as CoCa, mixed-modal generators such as Chameleon [Team, 2024] and CM3leon [Yu et al., 2023], and text-to-image models such as Stable Diffusion, Imagen, and Parti [Yu et al., 2022c].Source: §2.4 · p. 11

2.4.1 An example of learning a text generator: CoCa

• CoCa combines contrastive and generative losses, enabling multimodal understanding tasks such as VQA without additional multimodal-fusion adaptation.Its multimodal decoder conditions on image-encoder outputs and unimodal text-decoder representations, and pretraining uses ALIGN’s ∼1.8B images plus JFT-3B’s >29.5k labeled classes treated as alt-text.Source: §2.4.1 · p. 11

2.4.2 An example of multi-modal generative model: Chameleon and CM3leon

• CM3leon [Yu et al., 2023] uses interleaved image and text tokens with a decoder-only transformer, retrieval-augmented pretraining, and multitask instruction tuning for multimodal generation and editing.Chameleon [Team, 2024] instead uses fully token-based early fusion across images, text, and code, with architectural and training modifications that improve optimization stability and alignment at scale.Source: §2.4.2 · pp. 11–12

2.4.3 Using generative text-to-image models for downstream vision-language tasks

• Generative text-to-image models can perform classification and caption prediction without retraining by comparing conditional image likelihoods pθ(x | c) across candidate text prompts.Autoregressive models estimate likelihoods after image tokenization, while diffusion models estimate variational lower bounds from noise-prediction errors.Source: §2.4.3 · pp. 12–14
• Generative classifiers offer stronger out-of-distribution robustness, outperform CLIP on Winoground, align better with human judgments, and can improve prediction under online distribution shift.Diffusion-based classification remains impractical because it scales with the number of classes and may require hundreds or thousands of network evaluations per image, despite good accuracy.Source: §2.4.3 · p. 14

2.5 VLMs from Pretrained Backbones

• Pretrained backbones reduce VLM training demands by learning a mapping between existing language and visual representations instead of training complete models from scratch.Training from scratch can require hundreds to thousands of GPUs and hundreds of millions of image-text pairs, whereas pretrained components require fewer compute resources.Source: §2.5 · pp. 14–15

2.5.1 Frozen

• Frozen connects a vision encoder to a frozen 7-billion-parameter language model through a lightweight mapping network trained with text generation on Conceptual Captions.The model supports interleaved text and image embeddings, rapid adaptation, general-knowledge access, and visual-linguistic binding, although its performance was modest.Source: §2.5.1 · p. 15

2.5.2 The example of MiniGPT

• MiniGPT-4 aligns pretrained BLIP-2 visual representations with Vicuna using only a trained linear projection, requiring four A100 GPUs for about ten hours plus two training rounds.The first round used 20k steps and roughly 5M image-text pairs; instruction tuning then used 400 steps with curated data.Source: §2.5.2 · p. 15
• MiniGPT-5 [Zheng et al., 2023] extends MiniGPT-4 to interleaved text-image generation by mapping generative tokens into features for a frozen Stable Diffusion 2.1 model.It is supervised on downstream tasks including multimodal dialogue and story generation.Source: §2.5.2 · p. 15
• MiniGPT-v2 [Chen et al., 2023b] unifies captioning, visual question answering, and object grounding through task-specific identifiers, and shows strong understanding on visual-question-answering and grounding benchmarks.The identifiers help the model distinguish task instructions and learn efficiently.Source: §2.5.2 · p. 16

2.5.3 Other popular models using pretrained backbones

• Other pretrained-backbone VLMs, including Qwen-VL [Bai et al., 2023b] and BLIP-2 [Li et al., 2023e], use compact alignment modules to connect visual encoders with LLM input spaces.Qwen uses one-layer cross-attention to compress visual features into 256 tokens, while BLIP-2 trains a roughly 100–200M-parameter Q-Former over frozen visual and language models.Source: §2.5.3 · p. 16

3 A Guide to VLM Training

• Data quality is central to VLM training: datasets should be diverse, balanced, deduplicated, pruned for image-caption relevance, and improved through higher-quality captions and grounding.Scaling remains influential, but data curation can improve efficiency and mitigate memorization; Figure 2 also identifies grounding and alignment as key training considerations.Source: §3 · pp. 16–17, Figure 2 · p. 17

3.1 Training data

• Data pruning uses heuristics, pretrained-VLM alignment scores, or diversity and balancing strategies to remove low-quality pairs and construct more useful pretraining sets.DataComp evaluates filtering strategies across noisy web-data pools and 38 downstream tasks, while CLIPScore, T-MARS, and generative captioning models provide alignment-based ranking alternatives.Source: §3.1 · pp. 17–18
• Balanced sampling can broaden concept coverage, but ImageNet-based selection may bias generalization, web data remains long-tailed, and zero-shot performance depends strongly on downstream concepts represented during training.MetaCLIP limits samples per query to 20,000 to balance represented and under-represented concepts, yet CLIP variants still show uneven performance across visual concepts.Source: §3.1 · pp. 18–19

3.1.1 Improving the training data with synthetic data

• Synthetic captions and images can replace noisy labels or enable training without real datapoints, but caption diversity limits gains at scale.BLIP-based captions can outperform human-written captions when consistent and complete, while SynCLR and SynthCLIP use LLM-generated captions and text-to-image models to create training pairs.Source: §3.1.1 · p. 19

3.1.2 Using data augmentation

• Data augmentation improves representation learning through auxiliary self-supervision, while CLIP-rocket extends this signal cross-modally using weak and strong augmentations with separate projectors.The weak and strong representations are interpolated at inference, and separating projectors prevents overly invariant strong-augmentation representations from harming downstream tasks.Source: §3.1.2 · pp. 19–20

3.1.3 Interleaved data curation

• Interleaved datasets are curated either from naturally co-occurring web text and images or by retrofitting text corpora with contextually matched images.OBELICS preserves document structure through deduplication and HTML, image, and text filtering, whereas MMC4 uses CLIP similarity to pair internet images with text-only data at scale but with less contextual nuance.Source: §3.1.3 · p. 20

3.1.4 Assessing multimodal data quality

• Multimodal data quality can be assessed separately for text, images, and image-text alignment, but no holistic scalable evaluation method yet covers interleaved data.Existing approaches include textual-quality pruning, aesthetic scoring, and CLIP-family alignment evaluation; high-quality interleaved data is linked to optimal VLM performance.Source: §3.1.4 · p. 21

3.1.5 Harnessing human expertise: the power of data annotation

• Human annotations provide fine-grained visual-language supervision for complex scenes, but their cost and limited scale make them better suited to evaluation or fine-tuning than large-scale pretraining.Many existing datasets rely on older benchmarks, while DCI adds detailed annotations for images from SA-1B and illustrates the need for more diverse contemporary imagery.Source: §3.1.5 · p. 21

3.2.1 Using existing public software repositories

• Public repositories such as OpenCLIP and transformers implement most VLMs and support benchmarking and downstream comparisons of pretrained models.These tools provide a practical platform for evaluating and comparing different VLMs on given tasks.Source: §3.2.1 · p. 22

3.2.2 How many GPUs do I need?

• Training CLIP-like contrastive models from scratch can require over 500 GPUs and hundreds of thousands of dollars, but suitable data and masking can reduce this to 64 GPUs and about 10K USD.Using pretrained image or text encoders, or an LLM, makes mapping substantially less costly.Source: §3.2.2 · p. 22

3.2.3 Speeding up training

• Efficient training combines torch.compile, xformers, faster data loading through uncompressed files or FFCV, and token masking for large models.Uncompressed storage can cost more, but faster training may quickly offset the added storage expense; masking can also improve performance while reducing training time.Source: §3.2.3 · p. 22

3.2.4 Importance of other hyper-parameters.

• Image resolution, visual encoder capacity, and visual pretraining data most affect VLM performance, whereas modality-connection choices matter much less.A suitable mix of text-only, interleaved, and image-caption paired data achieves the best results across zero-shot classification and visual-question answering.Source: §3.2.4 · p. 23

3.3 Which model to use?

• VLMs can be trained contrastively, with masking, autoregressively, by diffusion, or through mappings between pretrained vision and language representations.The central choice is whether to train encoders jointly or connect pretrained backbones through a learned representation mapping.Source: §3.3 · p. 23

3.3.1 When to use contrastive models like CLIP?

• CLIP aligns image and text representations effectively for retrieval and grounding, but cannot generate captions and requires large datasets, batches, and training resources.CLIP is especially suitable as a foundation for testing additional objectives or architectures that improve relational and conceptual grounding.Source: §3.3.1 · p. 23

3.3.2 When to use masking?

• Masking jointly models image-text distributions without negative examples or batch dependence, but decoder-based reconstruction can introduce an efficiency bottleneck.Many VLMs combine masking strategies with contrastive losses.Source: §3.3.2 · pp. 23–24

3.3.3 When to use a generative model?

• Generative diffusion and autoregressive models directly decode likely images from text and learn implicit joint distributions, but cost more to train than contrastive models.Their outputs can make learned representations easier to inspect than retrieval pipelines requiring extensive k-NN evaluation.Source: §3.3.3 · p. 24

3.3.4 When to use LLM on pretrained backbone?

• Pretrained vision or language backbones reduce compute by requiring only representation mapping, but can transfer hallucinations and biases that require additional correction.This approach is most useful when resources are limited and the goal is mapping between pretrained representation spaces.Source: §3.3.4 · p. 24

3.4 Improving grounding

• Grounding seeks to prevent prompt omissions and hallucinations while improving understanding of relations, negation, counting, and visual attributes, but no single solution currently suffices.The section presents bounding-box supervision and negative samples as practical strategies for improving grounding.Source: §3.4 · p. 25

3.4.1 Using bounding boxes annotations

• Bounding-box annotations help align textual descriptions with visual regions; X-VLM uses box regression and IoU loss across up to 16 million images and multiple datasets.Automatically generated annotations, as in Kosmos-2, scale web-grounded data but inherit errors from the underlying grounding model, especially for rare nouns or instances.Source: §3.4.1 · p. 25

3.4.2 Negative captioning

• Negative samples improve VLMs by forcing finer distinctions between correct and incorrect image-caption pairings, producing more accurate and contextually aware models.The ARO benchmark evaluates this ability using negative samples that test associations with incorrect or nonsensical captions.Source: §3.4.2 · pp. 25–26

3.5 Improving alignment

• Instruction tuning adapts vision-language models using supervised instruction-input-response data, while RLHF uses a reward model trained on human preferences to align outputs.Instruction-tuning datasets typically contain a few to one hundred thousand samples, whereas RLHF fine-tunes the primary model using an auxiliary reward model.Source: §3.5 · p. 26

3.5.1 A LLaVA story

• LLaVA uses 150k synthetically generated visual instruction samples with Vicuna, CLIP ViT-L/14, and a linear projector, improving instruction following and Science QA.The model also improves qualitative multimodal chat interactions.Source: §3.5.1 · p. 26
• LLaVA 1.5 combines a cross-modal fully connected MLP with academic VQA instruction data, training on 600k image-text pairs in approximately one day on 8-A100 GPUs.It performs well on academic VQA and instruction-following benchmarks and is described as more training-efficient than InstructBLIP or Qwen-VL.Source: §3.5.1 · p. 27
• LLaVA-RLHF uses Factually Augmented RLHF, GPT4-generated data, and human-written image-text pairs to reduce hallucinations and improve factual multimodal alignment.It reaches 94% of GPT-4 performance on LLaVA-Bench and outperforms baselines by 60% on MMHAL-BENCH [Sun et al., 2023].Source: §3.5.1 · p. 27
• LLaVA-NeXT improves LLaVA-v1.5 through higher-resolution visual features, stronger reasoning and OCR data, and a 34B-parameter LLM backbone.It achieves state-of-the-art performance among open-source multimodal LLMs and closes the gap with Gemini Pro [Reid et al., 2024].Source: §3.5.1 · p. 27

3.5.2 Multimodal in-context learning

• Otter demonstrates multimodal in-context learning by following test instructions from contextual instruction-image-answer examples without extra fine-tuning.Fine-tuning OpenFlamingo on the 2.8M-pair MIMIC-IT dataset produces stronger instruction following and multimodal in-context learning.Source: §3.5.2 · pp. 27–28

3.6 Improving text-rich image understanding

• LLaVAR augments visual instruction tuning with OCR-derived data from 422K text-rich images and 16K GPT-4-generated conversations, improving text-based VQA accuracy by up to 20%.It also slightly improves performance on natural images, addressing MLLMs’ difficulty with complex relationships involving image text.Source: §3.6 · p. 28
• Monkey uses sliding-window patches, LoRA, a visual resampler, and multi-level descriptions for images up to 1344×896, while Lumos decouples on-device scene-text recognition from cloud multimodal reasoning.Lumos reconstructs reading order and processes images up to 3kx4k, reducing high-resolution image transfer power and latency.Source: §3.6 · pp. 28–29

Overview

• PEFT methods address the impractical cost of fine-tuning growing pretrained models by training parameter subsets across four families: LoRA-, prompt-, adapter-, and mapping-based methods.These approaches adapt models to downstream tasks without updating the entire parameter set.Source: §3.7 · p. 29

LoRA-based methods

• LoRA variants improve efficiency through quantization, shared low-rank parameters, or magnitude-direction decomposition while extending low-rank adaptation to vision-language benchmarks.QLoRA backpropagates through a frozen 4-bit quantized language model; VeRA uses shared matrices and scaling vectors; DoRA separates magnitude and direction.Source: §3.7 · pp. 29–30

Prompt-based methods

• Prompt-based methods learn prompt context or input-space tokens instead of fully tuning backbones, with CoOp outperforming handcrafted prompts and linear probes across 11 datasets.VPT trains less than 1% of model parameters with the backbone frozen and can match or exceed full-fine-tuning accuracy.Source: §3.7 · p. 30

Adapter-based methods

• Adapter methods add bottleneck modules or selectively unlock parameters, while weight sharing can rival full fine-tuning using only 4.18% of image-text and 3.39% of video-text parameters.CLIP-Adapter blends adapted and pretrained features, while LLaMA-Adapter V2 uses early visual-token fusion and additional learnable parameters for multimodal reasoning.Source: §3.7 · p. 30

Mapping-based methods

• Mapping-based methods keep pretrained vision encoders and language models frozen, training only a connector; MAPL aggregates visual features into query-token outputs to reduce sequence length versus LiMBeR.This architecture-independent approach uses fewer trainable parameters and improves data efficiency, whereas LiMBeR projects each visual feature independently into the language model space.Source: §3.7 · pp. 30–31

4.1 Benchmarking visio-linguistic abilities

• VLM evaluation uses captioning, VQA, zero-shot classification, compositional benchmarks, dense annotations, and synthetic data to test increasingly complex visual-linguistic mappings.These range from recognizing simple visual clues to spatial reasoning, caption-image alignment, and robustness to controlled variations.Source: §4.1 · p. 31, Figure 3 · p. 32

4.1.1 Image captioning

• Captioning metrics such as BLEU and ROUGE approximate similarity to reference captions, while CLIPScore measures image-caption alignment but inherits limitations from its underlying CLIP model.For text-to-image evaluation, VQA-based consistency scores, TIFA, DSG, VPEval, and VQAScore assess whether generated images satisfy textual descriptions; DSG propagates errors through scene-graph dependencies.Source: §4.1.1 · p. 33, §4.1.2 · p. 33

4.1.3 Visual question answering

• VQA supports broad image-question evaluation, but exact-string VQA Accuracy can underestimate free-form and out-of-distribution performance, motivating LLM judges and selective-prediction measures.Selective prediction evaluates coverage at a tolerated risk and Effective Reliability, with abstention based on answer probability, learned correctness, or model agreement.Source: §4.1.3 · pp. 33–34

4.1.4 Text-centric Visual Question Answering

• Text-centric VQA evaluates reading, structuring, and reasoning over scene text, documents, extracted key information, and handwritten mathematical expressions across task-specific benchmarks.Benchmark choice matters because datasets differ in multilingual, handwritten, non-semantic, mathematical, infographic, and tabular content.Source: §4.1.4 · pp. 34–35

4.1.5 Zero-shot image classification

• Zero-shot classification performance depends strongly on prompt construction and whether evaluated concepts appeared in training data, weakening claims that such tests are genuinely zero-shot.Distribution alignment can improve performance on some out-of-distribution benchmarks, but the underlying training distribution may remain unknown.Source: §4.1.5 · pp. 35–36

4.1.6 Visio-linguistic compositional reasoning

• Compositional benchmarks compare correct and perturbed captions to test relations, attributes, and word order, but procedurally generated negatives may be incoherent and tied scores can produce artificially perfect accuracy.Winoground uses paired images and captions, ARO swaps caption elements without true negative images, and SUGARCREPE uses more plausible ChatGPT-generated hard negatives.Source: §4.1.6 · pp. 36–37

4.1.7 Dense captioning and crop-caption matching

• Dense captioning, synthetic scenes, and crop-caption matching address limitations of short descriptions and biased real-image datasets by providing fine-grained or controlled evaluation; PUG finds current VLMs no better than chance on spatial relations.DCI annotates 7805 images with captions above 1000 words, while synthetic generation controls scene elements and supplies granular ground truth.Source: §4.1.7 · p. 37, §4.1.8 · p. 38

4.2 Benchmarking Bias and disparities in VLMs

• Responsible evaluation also measures demographic bias, language shortcuts, hallucination, and memorization, because benchmarks can be solved without visual grounding and models can generate confident false content.VQA contains a 98% “yes” prior for questions beginning “Is there a clock”; balanced benchmarks penalize unimodal shortcuts, while hallucination tests include CHAIR, POPE, model-based evaluation, and human annotations, and cross-modal memorization remains difficult to expose.Source: §4.2.1 · p. 39, §4.2.2 · p. 39, §4.2.3 · p. 40

4.4 Benchmarking memorization

• A k-nearest-neighbor comparison between a target CLIP model and an unseen reference model quantifies déjà vu memorization through object-detection precision/recall gaps [2024].A larger gap indicates greater memorization of objects present in training images but absent from captions.Source: §4.4 · p. 41
• Text randomization most effectively reduces memorization without severely penalizing utility by masking random caption tokens during each training epoch [2024].The augmentation weakens overfitting to associations between training captions and their corresponding images.Source: §4.4 · pp. 41–42

4.5 Red Teaming

• Red teaming probes public VLM interfaces with adversarial prompts and datasets to elicit and score undesirable outputs [Perez et al., 2022; Vidgen et al., 2023, Bianchi et al., 2024].Reference answers, such as refusals, provide a basis for scoring model responses.Source: §4.5 · p. 42
• VLM red teaming must address multimodal harms, including faithfulness, privacy, safety, and fairness, such as harmful graphic descriptions elicited from sensitive images [Li et al. 2024b].Existing language and text-to-image evaluations organize harms into risk taxonomies and adversarial leaderboards, while formal risk definitions remain under active research [Weidinger et al., 2022, Sun et al., 2024, Derczynski et al., 2023; Liang et al., 2022, Röttger et al., 2024; Lee et al. 2024].Source: §4.5 · p. 42
• Red-team findings can support risk mitigation through post-processing or model fine-tuning, including Reinforcement Learning for Human Feedback.Source: §4.5 · p. 42

5 Extending VLMs to Videos

• Video introduces motion and spatiotemporal localization capabilities but increases storage, memory, and training costs roughly with frame rate, motivating compression, pooling, masking, and offline features.A 24 fps video can require 24× the storage and processing of treating each frame as an image.Source: §5 · pp. 42–43

5.1 Early work on Videos based on BERT

• VideoBERT fused frame and ASR tokens with a transformer and masked-token reconstruction, achieving strong alignment and text-generating video capabilities.It performed well on zero-shot action classification and open-ended video captioning using instructional cooking videos.Source: §5.1 · p. 43

5.2 Enabling text generation using an early-fusion VLM

• VideoOFA [Chen et al., 2023c] adapted a generative image-text VLM through video captioning, matching, and frame-order objectives, improving video captioning and question answering.The intermediate pretraining stage teaches video-specific concepts such as temporal reasoning while retaining a text-generating backbone.Source: §5.2 · p. 44

5.3 Using a pretrained LLM

• MiniGPT4-Video outperformed Video-LLaMA [Zhang et al., 2023b] by a large margin on MSVD, MSRVTT, TGIF, and TVQA benchmarks.Video-LLaMA aligns separate video and audio Q-formers with an LLaMA model and instruction-tunes it for conversational interaction.Source: §5.3 · pp. 44–45

5.4 Opportunities in evaluations

• Synthetic physics evaluations reveal that VideoLLaMA and PandaGPT [Su et al., 2023] remain at random performance, while humans exceed 80% accuracy, exposing weak temporal reasoning.Video benchmarks also test long-video interactions, relevant-frame retrieval, and action localization, although some questions can be answered from one frame.Source: §5.4 · p. 45

5.5 Challenges in leveraging video data

• Video VLMs are limited by scarce temporal supervision, noun bias, costly video-caption creation, expensive computation, and redundant frames, requiring better data and efficient training protocols.Scene-focused internet captions can make video models behave like image models; proposed remedies include generated captions, video-only encoder training, and masking.Source: §5.5 · pp. 45–46

Training VLMs

• High compute and data costs motivate leveraging pretrained LLMs or image encoders to learn mappings between modalities.VLMs can be trained using contrastive or generative methods, but resource demands remain a barrier for many researchers.Source: §6 · p. 46

Reliability

• Large-scale high-quality images and captions, stronger grounding, and alignment with human preferences are needed to improve VLM reliability.These are identified as general considerations regardless of the training technique.Source: §6 · p. 46

Evaluation

• Existing benchmarks assess vision-linguistic and reasoning abilities but can be severely limited by solvability through language priors.Benchmark design remains an important challenge for measuring genuine multimodal performance.Source: §6 · p. 46

Video

• Video is an important modality for learning representations, yet substantial challenges remain before high-quality video representations can be learned.VLM research therefore extends beyond binding images to text.Source: §6 · p. 46

Models and objectives

• Major vision-language models and objectives include CLIP (Contrastive Language–Image Pre-training), BLIP (Bootstrapping Language-Image Pre-training), CoCa (Contrastive Captioner), FLAVA (Foundational Language And Vision Alignment), and VQ-VAE (Vector Quantised-Variational AutoEncoder).Source: Acronyms · pp. 47–48

Datasets and resources

• Datasets and multimodal resources include OBELICS (Open Bimodal Examples from Large fIltered Commoncrawl Snapshots), DCI (Densely Captioned Images), PUG (Photorealistic Unreal Graphics), and MERLOT (Multimodal Event Representation Learning Over Time).Source: Acronyms · pp. 47–48

Training and adaptation

• Training and adaptation terms include PEFT (Parameter-Efficient Fine-Tuning), LoRa (Low Rank Adapters), RLHF (Reinforcement Learning from Human Feedback), and FFCV (Fast Forward Computer Vision).Source: Acronyms · pp. 47–48

Learning and optimization

• Learning and optimization abbreviations include SSL (Self-Supervised Learning), MLM (Masked Language Modeling), MIM (Masked Image Modeling), NCE (Noise Contrastive Estimation), EBM (Energy-Based Models), and MCMC (Markov Chain Monte Carlo).Source: Acronyms · pp. 47–48

Modeling and architectures

• Core modeling terms include VLMs (Vision Language Models), MLLMs (Multimodal Large Language Models), LLMs (Large Language Models), ViT (Vision Transformer), and CNN (Convolutional Neural Network).Source: Acronyms · pp. 47–48

Evaluation and applications

• Evaluation and application terms include VQA (Visual Question Answering), IoU (Intersection over Union), ARO (Attribution, Relation, and Order), OOD (Out-Of Distribution), ROI (Region of Interest), and STR (Scene text recognition).Source: Acronyms · pp. 47–48