Source-linked AI summary
What matters when building vision-language models?
Hugo Laurençon, Léo Tronchon, Matthieu Cord, Victor Sanh
TL;DR
VLM research often leaves important design choices insufficiently justified, limiting clarity about which choices improve performance. The paper addresses this gap with controlled experiments across backbones, architectures, data, and training methods, and develops Idefics2. The resulting 8B model achieves state-of-the-art performance in its size category and is sometimes comparable to models four times larger.
Problem
Critical VLM design decisions are often not justified, making it difficult to identify which choices improve model performance.
Method
The authors rigorously compare pre-trained backbones, architectures, connector modules, data, and training methods in controlled experiments, then train Idefics2 from the findings.
Results
Idefics2 is an 8B VLM with state-of-the-art performance in its size category across various benchmarks, sometimes matching models four times larger.
Takeaways & Limitations
The paper provides experimentally grounded findings on VLM architecture, training stability, and inference-efficiency trade-offs, alongside released models and training data.
Takeaways & Limitations
The open VLM community lacks a large, well-trained vision encoder, limiting conclusions about the potential of larger vision backbones.
Abstract
from arXiv · showhide
The growing interest in vision-language models (VLMs) has been driven by improvements in large language models and vision transformers. Despite the abundance of literature on this subject, we observe that critical decisions regarding the design of VLMs are often not justified. We argue that these unsupported decisions impede progress in the field by making it difficult to identify which choices improve model performance. To address this issue, we conduct extensive experiments around pre-trained models, architecture choice, data, and training methods. Our consolidation of findings includes the development of Idefics2, an efficient foundational VLM of 8 billion parameters. Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks, and is often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.
1 Introduction
The paper argues that poorly justified VLM design choices make it difficult to determine what drives performance and to make grounded progress. It addresses this through controlled comparisons of architectures, training procedures, and related efficiency choices, then uses the findings to build Idefics2.
- Unjustified design decisions make it difficult to distinguish which choices account for VLM performance.
- The study compares model architectures, connector modules, multimodal training procedures, and their effects on inference efficiency and training stability.
- The authors find that progress is driven substantially by pre-trained unimodal backbones, while fully autoregressive architectures outperform cross-attention architectures after training modifications.
- Idefics2 is an 8 billion-parameter VLM that achieves state-of-the-art performance in its size category across various benchmarks while being more efficient at inference.
- Idefics2 is on par with state-of-the-art models four times larger on some vision-language benchmarks and matches Gemini 1.5 Pro on some challenging benchmarks.
2 Terminology
VLMs connect pre-trained vision and language backbones through newly initialized parameters and commonly train them in pre-training followed by instruction fine-tuning. The terminology distinguishes cross-attention from fully autoregressive fusion, in which visual and text tokens form one language-model input sequence.
- VLM training typically connects pre-trained vision and language backbones by initializing new parameters between the two modalities.
- Pre-training commonly uses large multimodal datasets, followed by instruction fine-tuning on task-oriented samples.
- Cross-attention architectures inject vision features through cross-attention blocks, whereas fully autoregressive architectures concatenate visual tokens with text tokens before language-model processing.
- In the fully autoregressive architecture, visual tokens can be pooled into a shorter sequence to improve compute efficiency.
3 Exploring the design space of vision-language models
The experiments compare pretrained backbones, multimodal architectures, training methods, pooling, and image-resolution strategies under controlled settings. They find that backbone quality, trainability, learned pooling, and flexible image processing jointly shape VLM performance and efficiency.
- Pre-trained backbones: Across fixed-size backbone comparisons, language-model quality has a higher impact on final VLM performance than vision-backbone quality.Replacing LLaMA-1-7B with Mistral-7B yields a 5.1-point benchmark boost, while switching vision encoders yields a 3.3 increase.
- Architecture and training: The cross-attention architecture performs better with frozen unimodal backbones, but the fully autoregressive architecture wins when those backbones are trained.Unfreezing produces a 12.9-point increase for fully autoregressive training versus 0.6 point for cross-attention training.
- Architecture and training: Unfreezing backbones under the fully autoregressive architecture can cause divergences, whereas LoRA adds training expressivity and stabilizes optimization.The authors select the fully autoregressive architecture for the remainder of the work.
- Efficiency: Learned pooling raises average performance by 8.5 points while reducing visual tokens per image from 729 to 64.The resulting shorter sequences improve compute efficiency during both training and inference.
- Efficiency: Using more than 64 visual tokens provides no observed gains, while other Perceiver variations decrease performance.The authors note that more tokens might help with unlimited training and data, but at the cost of longer training time.
- Efficiency: Preserving original image aspect ratio and resolution maintains downstream performance while speeding training and inference and reducing memory use.Splitting images into sub-images can trade training efficiency for better inference performance, especially for reading text in images.
4 Idefics2 - an open state-of-the-art vision-language foundation model
Idefics2 is an open 8B-parameter VLM built from controlled design findings, diverse pre-training data, and staged instruction tuning. It performs strongly across multimodal benchmarks while remaining efficient at inference.
- Model construction: Idefics2 is an open 8B-parameter VLM constructed from the paper’s findings on model design, data, and training phases.The model starts from SigLIP-SO400M and Mistral-7B-v0.1.
- Pre-training data: The pre-training mixture combines interleaved image-text documents, image-text pairs, and PDF documents.OBELICS supplies long interleaved documents; PMD and web-scale datasets supply image-text pairs; OCR-IDL and PDFA supply PDF data.
- Training procedure: The model uses two compute-oriented pre-training stages, beginning with 384-pixel images and later introducing PDF documents.The first stage uses a global batch size of 2’048 and samples OBELICS at 70% of examples.
- Instruction data: 50 vision-language datasets in The Cauldron cover tasks including visual question answering, counting, captioning, OCR, document understanding, and reasoning.The instruction mixture also includes text-only datasets for complex instruction following, mathematics, and arithmetic.
- Training procedure: Idefics2 is instruction-tuned with DoRA, answer-token-only loss, embedding noise, randomized image resolution, and additional dialogue training.The dialogue phase addresses the tension between short benchmark answers and longer conversational generations.
- Evaluation: Idefics2 outperforms current best foundation VLMs in its size category and is on par with models four times larger or Gemini 1.5 Pro on several benchmarks.The reported comparisons cover MMMU, MathVista, TextVQA, and MMBench, with computationally much more efficient inference.
5 Conclusion
The paper combines controlled comparisons of VLM design choices with the training and release of Idefics2, an open 8B-parameter model. Idefics2 is reported as state-of-the-art in its size category and efficient at inference.
- Conclusion: Controlled experiments compare VLM architectures, performance–inference-cost trade-offs, and training stability.The authors re-examine common choices and report findings from controlled comparisons.
- Conclusion: Idefics2 is an open 8B-parameter VLM that is state-of-the-art across various benchmarks in its size category and much more efficient at inference.The paper releases its findings, models, and training dataset.
A.1.1 Cross-attention vs. fully autoregressive architectures
The fully autoregressive architecture improves average performance across evaluation axes compared with the cross-attention architecture. The comparison considers steps, image count, and text-token count.
- Architecture comparison: The fully autoregressive architecture improves average performance across the number of steps, number of images, and number of text tokens.The authors report improvement across the board in Figure 4.
- Architecture comparison: Figure 4 compares cross-attention and fully autoregressive architectures using steps, images, and text tokens as evaluation axes.Average scores across these axes are used for the comparison.
A.1.2 Comparing various vision backbones
The vision-backbone ablation compares multiple encoders and highlights a parameter-efficiency difference between EVA-CLIP-5B and SigLIP-SO400M. TextVQA is especially sensitive to image resolution.
- Vision backbone comparison: EVA-CLIP-5B performs similarly to SigLIP-SO400M despite having 11 times more parameters.The comparison is reported in the detailed vision-backbone ablation.
- Vision backbone comparison: TextVQA is the most sensitive benchmark to image resolution among the early experiments.This sensitivity accounts for the reported performance increase.
A.1.3 Comparing various pooling strategies
The perceiver resampler outperforms the tested linear and Mapping Network pooling strategies. Increasing its depth shows no statistically significant benefit, while adding modality projection before it improves performance.
- The perceiver resampler significantly outperforms both the flattened-sequence linear layer and the Mapping Network.These alternatives were evaluated as pooling strategies for shortening visual-token sequences.
- Increasing the number of perceiver resampler layers produces no statistically significant performance differences.The authors select 3 layers cautiously to avoid a potential capacity bottleneck.
- A 2-layer modality projection MLP before the perceiver resampler yields better performance.The MLP projects the vision encoder’s hidden dimension to the language model’s hidden dimension.
A.1.4 Ablations on OCR data
Adding OCR documents did not produce statistically significant zero- or few-shot differences, but fine-tuning on DocVQA revealed stronger checkpoint differences. The broader evaluation also notes that open-ended VQA metrics can penalize semantically equivalent answers, while red-teaming exposes safety and security concerns.
- A.1.4 Ablations on OCR data: OCR-document training produced no statistically significant differences in zero- or few-shot evaluation.The comparison also controlled image legibility by increasing image resolution.
- A.1.4 Ablations on OCR data: Fine-tuning checkpoints on DocVQA for 500 steps at a 1e −5 learning rate produced much stronger differences.
- A.3.2 Expanded evaluation table: Open-ended VQA metrics can count synonymous or more verbose answers as incorrect, limiting the interpretability of small score gaps.Manual analysis found that a 5-point difference on benchmarks such as VQAv2 could be barely noticeable.
- A.4 Red-teaming: The model can hastily form judgments requiring nuanced contextual understanding and perpetuate harmful stereotypes.Reported examples include judgments based on visual cues, offensive content from benign images, inferred mental states, and attractiveness assessments.
- A.4 Red-teaming: The model can solve CAPTCHAs with distorted text, increasing existing security risks.
- A.4 Red-teaming: Repeated or guided interactions can jailbreak cautionary prompts, creating risks that require critical thinking and discretion.The red-teaming discussion identifies additional visual avenues for malicious prompts and interactions between vision and language vulnerabilities.