Source-linked AI summary
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, Dorsa Sadigh
TL;DR
VLM design choices are under-explored, and inconsistent evaluations make their effects difficult to assess. The paper addresses this by building standardized evaluations, examining four design axes with flexible training infrastructure, and releasing checkpoints. Its PRISMs outperform state-of-the-art open VLMs, while the study remains bounded by its architecture and evaluation scope.
Problem
Existing VLM approaches cover only a sliver of the design space, while objective, diverse evaluations for capabilities such as spatial reasoning and hallucination are needed.
Method
The paper compiles twelve standardized benchmarks, investigates four VLM design axes, and provides optimized open training code and model checkpoints.
Results
PRISMs outperform state-of-the-art open VLMs such as InstructBLIP and LLaVa-v1.5 across the study’s evaluated capabilities.
Takeaways & Limitations
The central contribution is a foundation for future VLM research through fine-grained evaluation and flexible, extensible training resources.
Takeaways & Limitations
The findings may not generalize to architectures outside the studied three-component design or to larger scales such as 70B+ parameters.
Abstract
from arXiv · showhide
Visually-conditioned language models (VLMs) have seen growing adoption in applications such as visual dialogue, scene understanding, and robotic task planning; adoption that has fueled a wealth of new models such as LLaVa, InstructBLIP, and PaLI-3. Despite the volume of new releases, key design decisions around image preprocessing, architecture, and optimization are under-explored, making it challenging to understand what factors account for model performance $-$ a challenge further complicated by the lack of objective, consistent evaluations. To address these gaps, we first compile a suite of standardized evaluations spanning visual question answering, object localization, and challenge sets that probe properties such as hallucination; evaluations that provide fine-grained insight VLM capabilities. Second, we rigorously investigate VLMs along key design axes, including pretrained visual representations and training from base vs. instruct-tuned language models, amongst others. We couple our analysis with three resource contributions: (1) a unified framework for evaluating VLMs, (2) optimized, flexible training code, and (3) checkpoints for all models, including a family of VLMs at the 7-13B scale that strictly outperform InstructBLIP and LLaVa v1.5, the state-of-the-art in open VLMs.
1. Introduction
Prismatic VLMs investigate under-explored design choices through standardized evaluations, targeted experiments, and open training resources. The work examines how these choices influence downstream capabilities across diverse tasks.
- Headline result: When controlling for data and scale, Prismatic models outperform LLaVa v1.5 across 12 diverse tasks while saving more than 30% of training compute.The comparison is summarized in Figure 1.
- Motivation: Existing VLM approaches cover only a sliver of the design space, motivating systematic study of choices that influence downstream capabilities.The paper emphasizes the need for diverse, objective tasks probing spatial reasoning, out-of-distribution generalization, and commonsense understanding.
- Contributions: The authors compile twelve standardized benchmarks spanning visual question answering, object localization, and challenge tasks.The challenge tasks evaluate fine-grained spatial reasoning, hallucination, and diagram understanding.
- Contributions: The study explores optimization, image processing and visual representations, language models, and scaling training time and data.These axes are investigated through targeted experiments designed to connect individual choices with downstream performance.
- Resources: The authors provide an optimized, modular training codebase that supports swapping pretrained components, optimization procedures, and data.The codebase and evaluation suite are released alongside checkpoints for models trained in the study.
2. Preliminaries
The paper defines a common VLM architecture that transforms image features into language-model inputs and trains it with next-token prediction. Its flexible implementation supports controlled experiments and reproduces LLaVa v1.5 more efficiently.
- Model Architecture: A VLM combines a visual representation backbone, a vision-language projector, and a language model.The backbone produces image features, the projector maps them into language-model embedding space, and the language model generates text.
- Model Architecture: The backbone maps an input image to a sequence of patch features, which the learned projector converts into language-model embeddings.The projected image sequence is concatenated with text-prompt embeddings before language-model decoding.
- Optimization: Training minimizes negative log-likelihood of the target output conditioned on the image and prompt.The parameters of the visual backbone, projector, and language model are optimized by gradient descent.
- Training Implementation: The training codebase is modular and supports arbitrary optimization procedures, interchangeable backbones, and reproducible implementation practices.It uses PyTorch, FSDP, and BF16 mixed precision to improve flexibility and hardware portability.
- Verification: On identical hardware, the FSDP-backed implementation achieves 20% faster step times than the reference LLaVa v1.5 training implementation.The result comes from an apples-to-apples reproduction at both 7B and 13B parameter scales.
3. Evaluation Suite
The evaluation suite uses established benchmarks with defined metrics to measure VLM capabilities across question answering, localization, and targeted challenge tasks. This breadth enables fine-grained analysis of individual design choices.
- Evaluation Principles: The unified suite prioritizes well-defined metrics over subjective automated judgments to provide fine-grained capability measurements.The authors use standardized evaluations to make the impact of design changes easier to assess.
- Visual Question Answering: Visual question answering benchmarks cover general visual reasoning, unanswerable questions, spatial reasoning, and text understanding.The benchmarks are VizWiz, VQAv2, GQA, and TextVQA.
- Localization: Localization benchmarks evaluate bounding-box prediction for short spatial descriptions, appearance-only descriptions, long descriptions, and cluttered scenes.The suite includes RefCOCO, RefCOCO+, RefCOCOg, and OCID-Ref, with OCID-Ref probing out-of-distribution generalization.
- Challenge Sets: Challenge sets probe spatial relationships, object counting, hallucination propensity, and scientific diagram understanding.The tasks are VSR, TallyQA, POPE, and AI2D; VSR is especially difficult, with most existing models failing to exceed the 51% majority-class baseline.
- Protocols: The evaluation protocols use validation splits for most benchmarks, with specified alternatives for GQA, VSR, and POPE.GQA uses test-dev, VSR uses zero-shot test, and POPE has a single evaluation split.
4. Experiments – Investigating Design Axes
The experiments probe optimization, visual processing, language-model choice, and scaling, revealing design choices that affect performance, efficiency, safety, and compute cost.
- Optimization Procedure: Single-stage training improves aggregate performance while saving 20-25% of training cost by removing explicit projector pretraining.The authors consequently use single-stage training in subsequent experiments.
- Optimization Procedure: Finetuning the visual backbone significantly degrades performance, especially on fine-grained localization tasks such as RefCOCO and OCID-Ref.This degradation occurs in both single- and multi-stage training paradigms.
- Image Processing & Visual Representations: Naive resizing generally outperforms letterbox padding, while cropping is suboptimal; scaling resolution to 336px or 384px significantly improves performance.For SigLIP, naive resizing and letterbox padding perform similarly.
- Image Processing & Visual Representations: Doubling image resolution quadruples the number of input patches and increases Transformer attention time complexity sixteen-fold at fixed patch granularity.The corresponding memory requirements also increase substantially.
- Scaling Properties: Training Time & Data: Training for two epochs significantly improves performance over one epoch, while adding LVIS-Instruct-4V and LRV-Instruct improves performance, with LRV-Instruct having the larger impact.The results indicate underfitting at one epoch and emphasize image diversity when scaling data.
5. PRISM – Distilling Key Insights
The paper distills design insights that simplify VLM training and improve downstream performance, then combines them into PRISM models that outperform open baselines across the evaluation suite.
- Image Processing and Visual Representations: Fused DINOv2 and SigLIP backbones with high-resolution images and naive resizing yield strong performance.
- Language Models: Base LMs such as Llama-2 match or exceed instruct-tuned LMs, while language-only co-training is important for safety.
- Scaling Properties: Adding diverse data and extending training time significantly boost performance.
- PRISM Models: PRISM models at the 7B and 13B scales uniformly outperform InstructBLIP and LLaVa v1.5 by large margins across the evaluation suite.The comparison includes PRISM (Controlled), trained with the same data and budget as LLaVa v1.5.
6. Limitations & Future Work
The study’s findings are bounded by its architecture coverage and evaluation choices, leaving generalization to larger models and richer interactions for future work.
- Architecture Scope: The architecture study does not capture innovations such as Perceiver-based patch downsampling used for interleaved image-text training.The authors note open questions about how findings generalize, especially at scales of 70B+ parameters.
- Evaluation Scope: The evaluation focuses on standardized objective metrics and therefore does not capture extended, topic-shifting visual dialogues.Future work will extend evaluation to longer, richer contexts and co-design VLMs with downstream applications.
7. Conclusion
The paper rigorously investigates the VLM design space and provides evaluation and training resources, using the resulting insights to train PRISMs that outperform state-of-the-art open VLMs.
- Contributions: The study investigates VLM design choices while contributing a fine-grained evaluation suite and an optimized, extensible training codebase.The authors frame these resources as a foundation for future VLM training and evaluation research.
- Results: PRISM models outperform state-of-the-art open VLMs such as InstructBLIP and LLaVa-v1.5.
Impact Statement
The paper makes its VLM resources openly accessible while documenting safety, bias, and reliability risks that remain in these systems. It also emphasizes extending evaluations to probe undesirable behaviors.
- Open Resources: Open training code, evaluation code, and model checkpoints are released to improve accessibility for the broader research community.The authors frame openness as beneficial for transparency, risk mitigation, and positive use of VLMs.
- Known Risks: VLMs inherit risks and biases from both language models and underlying vision models and their pretraining datasets.
- Toxic and Unsafe Outputs: Safety tuning can reduce the ease of generating toxic content at minimal performance cost, but it does not prevent unsafe outputs in all settings.The authors report particular vulnerability to adversarial and out-of-distribution images.
- Western Bias & (American) English Bias: The paper identifies heavy American English and Western cultural bias, with limited visual-data diversity because pretraining images primarily come from English Flickr subsets.
- Factuality, Hallucination, & Reliability: The evaluation suite includes VizWiz and POPE to probe reliability and hallucination, including unanswerable questions and adversarially co-occurring objects.
- Training and Finetuning Accessibility: Efficient training lowers the barrier to experimentation: fully training a 7B VLM takes less than 9 hours on 8 A100 GPUs, while finetuning and evaluation can run on individual GPUs or CPU.
- Extending our Evaluation Suite: The evaluation suite is designed to support adding new VLMs and tasks, especially evaluations of bias, toxicity, hallucination, and other unsafe behaviors.
A.1. Pretraining Dataset Composition
The training data uses a two-stage pipeline: vision-language alignment followed by multimodal instruction tuning. The instruction mixture combines synthetic conversations, standard and multiple-choice VQA, captioning, grounding, and language-only data.
- Training Pipeline: Training has two stages: vision-language alignment freezes the visual representation and language model while training projector Fψ, followed by multimodal instruction tuning that trains Fψ and the LM.
- Vision-Language Alignment: The alignment subset uses 558K captioning examples from LAION, Conceptual Captions, and SBU, augmented with BLIP-generated captions.
- Multimodal Instruct Tuning: The multimodal instruction-tuning subset contains 665K examples designed to induce chat-like behavior and task-specific outputs through trigger prompts.
- Multimodal Instruct Tuning: The mixture includes 158K synthetic conversations and descriptions generated from COCO captions and bounding boxes by prompting GPT-4.
- Multimodal Instruct Tuning: Standard VQA data contributes 224K examples spanning general, spatial, external-knowledge, and OCR-focused question answering.
- Multimodal Instruct Tuning: Additional components include 50K multiple-choice VQA examples, 22K captioning examples, and 116K referring-expression grounding and region-captioning examples.
- Data Formats: ShareGPT contributes 40K language-only conversations, while localization uses normalized bounding-box coordinates generated as natural-language strings.
A.2. Implementation – Architecture Components & Optimization
The implementation combines pretrained vision backbones, a lightweight projector, and language models through sequence concatenation. Training uses PyTorch with FSDP and BF16 mixed precision, while prompting differs for base and instruct-tuned LMs.
- Optimization: The codebase uses PyTorch, Fully Sharded Data Parallel, and BF16 mixed precision to distribute training across GPUs.
- Architecture Components: Image processing uses torchvision and TIMM transforms, including resizing schemes and backbone-specific pixel normalization.
- Architecture Components: All evaluated visual representations use Vision Transformers, with patch features extracted from the penultimate layer.
- Architecture Components: The vision-language projector is a simple 2-layer GELU MLP that independently maps each patch into the language model’s embedding space.
- Architecture Components: Projected patch embeddings are concatenated before language-prompt embeddings, functioning similarly to prefix tuning.
- Prompting: Instruct-tuned LMs use a system prompt with formatted USER and ASSISTANT blocks, whereas base LMs omit the system prompt.
A.3. Training Hyperparameters
The paper fixes common hyperparameters across single-stage experiments and specifies evaluation procedures for reproducible comparison. Deterministic decoding, task-specific prompts, task metrics, and normalized global scores structure the analysis.
- Training Hyperparameters: Single-stage experiments use the Table 1 hyperparameters for both 7B and 13B models, while projector alignment uses batch size 256 and learning rate 1e-3.
- Evaluation Procedure: Evaluation procedures define task prompting, metric computation, and statistical-significance analysis, with corresponding procedures exposed in the evaluation codebase.
- Response Generation: Greedy decoding makes outputs deterministic and comparisons consistent, but may produce lower-quality outputs than nucleus sampling or beam search.
- Prompting: Task-specific trigger prompts are used to induce the output structures expected by each evaluation and to compare models consistently.
- Metrics: Open-ended VQA is scored with official accuracy scripts, with an OCR-prompted TextVQA variant reported only in the appendices for comparison with prior work.
- Metrics: Localization accuracy uses 0.5 IoU for RefCOCO variants and 0.25 IoU for OCID-Ref.
- Metrics: Challenge tasks are formatted as multiple-choice questions and scored by accuracy, using two, four, or sixteen options depending on the task.
- Aggregate Analysis: Normalized Z-scores across all models support global scores obtained by averaging performance over all 12 benchmarks.
B.3. Exhaustive Results
The paper tabulates evaluation results for all models trained in this work across VQA, localization, and challenge benchmarks.
- Open-ended VQA results for all trained models are tabulated in Table 2.
- Localization results for all trained models are tabulated in Table 3.
- Challenge Set results for all trained models are tabulated in Table 4.