Source-linked AI summary
Building and better understanding vision-language models: insights and future directions
Hugo Laurençon, Andrés Marafioti, Victor Sanh, Léo Tronchon
TL;DR
VLM development lacks consensus on architecture, data, training, and evaluation choices. This paper surveys those choices and details an efficient open-data construction of Idefics3-8B, whose Docmatix-supported training yields substantial document-understanding gains. It also identifies benchmark contamination and format sensitivity as evaluation concerns.
Problem
VLM research lacks consensus on architecture, data, training, and evaluation choices, while challenging capabilities such as document understanding remain limited by small open datasets.
Method
The paper provides a tutorial and survey of VLM development, then describes building Idefics3-8B with a straightforward, efficient pipeline using open datasets, including Docmatix.
Results
13.7 points: Idefics3-8B improves over Idefics2-8B on DocVQA, with Docmatix providing 2.4 million images and 9.5 million QA pairs from 1.3 million PDFs.
Takeaways & Limitations
The paper offers an open model and training datasets while identifying architecture, data, training, and evaluation choices as central development considerations.
Takeaways & Limitations
At least 6.1% of MathVista questions ask variants of a recurring age-gap question, giving models exposed to related training data an advantage.
Abstract
from arXiv · showhide
The field of vision-language models (VLMs), which take images and texts as inputs and output texts, is rapidly evolving and has yet to reach consensus on several key aspects of the development pipeline, including data, architecture, and training methods. This paper can be seen as a tutorial for building a VLM. We begin by providing a comprehensive overview of the current state-of-the-art approaches, highlighting the strengths and weaknesses of each, addressing the major challenges in the field, and suggesting promising research directions for underexplored areas. We then walk through the practical steps to build Idefics3-8B, a powerful VLM that significantly outperforms its predecessor Idefics2-8B, while being trained efficiently, exclusively on open datasets, and using a straightforward pipeline. These steps include the creation of Docmatix, a dataset for improving document understanding capabilities, which is 240 times larger than previously available datasets. We release the model along with the datasets created for its training.
1 Introduction
VLM research lacks consensus on key pipeline choices, so this paper reviews architectures, data, training, and evaluation before presenting an efficient open-data build of Idefics3-8B. Idefics3-8B substantially improves document understanding, supported by the large Docmatix dataset.
- VLMs support applications including document and figure understanding, visual mathematics, and screenshot-to-code generation.
- Divergent, often unjustified choices in architectures and other pipeline stages make it difficult to determine which decisions affect model performance.
- The paper surveys VLM architectures, training data, multi-stage training methods, evaluation challenges, and future research directions.
- 13.7 points: Idefics3-8B improves on Idefics2-8B for DocVQA while using an efficient pipeline and only open datasets.
- Docmatix contains 2.4 million images and 9.5 million QA pairs from 1.3 million PDF documents, a 240-fold increase over previous open datasets.
2 Analyzing architectural choices in VLMs
VLMs commonly connect pretrained vision and language backbones through self-attention or cross-attention, with performance depending on training conditions and backbone quality. The surveyed alternatives expose efficiency, context, evaluation, and scaling trade-offs.
- Cross-attention architecture: Most VLMs build on pretrained language and vision backbones, while cross-attention inserts vision-conditioned layers between language-model blocks.
- Self-attention architecture: Self-attention treats vision-encoder outputs as visual tokens, optionally pools them, concatenates them with text tokens, and feeds the combined sequence to the language model.
- Architecture comparison: Cross-attention outperforms self-attention with frozen backbones, but performs worse after LoRA training adds 200M trainable parameters despite having more parameters overall.
- Pretrained backbones: Replacing either language or vision backbones with stronger unimodal models improves VLM performance without changing the total parameter count.
- Vision encoders: Combining multiple vision encoders can create richer visual embeddings but reduces computational efficiency, while the field still needs extensively trained open vision encoders at scale.
- Vision encoder necessity: Vision-encoder-free designs preserve raw image information and avoid another pretrained model, but have not demonstrated superior benchmark performance and may require longer training.
- Image resolution and token efficiency: Variable-resolution tiling allocates fewer visual tokens to simpler tasks and more computation to high-resolution OCR, but separately encoded tiles can lose global context.
- Future directions: A proposed future direction is a vision encoder that natively handles varying, very large resolutions while preserving aspect ratios and efficiently managing long context.
3 Training methods and datasets for VLMs
VLM training typically uses multiple stages that progress from broad alignment data toward higher-quality, task-specific examples. The paper surveys dataset types and training phases, emphasizing their complementary roles and limitations.
- Multi-stage training: Training progresses through stages because data availability, memory, and stability constraints require gradual increases in data quality, image resolution, and trainable model components.The process begins with pretrained language and vision backbones.
- Multi-stage training: Pre-training aligns the backbones and newly initialized parameters using large-scale data intended to build broad knowledge and robustness.Some approaches initially freeze the backbone models to preserve the language model’s performance.
- Pre-training datasets: Image-text pairs are abundant and effective for alignment, but web alt-text is often noisy, ungrammatical, or too brief; synthetic recaptioning improves caption quality.LAION COCO recaptioned 600 million images, while VeCap contains 300 million samples combining original and synthetic captions.
- Pre-training datasets: Interleaved image-text documents preserve webpage ordering and can improve in-context learning, multi-image understanding, and text-distribution coverage.OBELICS contains 141 million documents and 353 million images after filtering Common Crawl HTML.
- Pre-training datasets: PDF datasets provide OCR-derived text and locations for document training, while synthetic data targets harder tasks that web-scale datasets underrepresent.Synthetic examples can address document understanding and visual math reasoning more directly than generic pre-training data.
- Fine-tuning: Fine-tuning commonly follows supervised fine-tuning with an alignment phase that targets instruction following, human preferences, reduced hallucination, and safety.SFT datasets combine many relatively small, high-quality datasets spanning diverse images and tasks.
4 Challenges in evaluating VLMs
Evaluating VLMs is difficult because open-ended benchmarks can reward answer-format matching, while pre-training scores may underestimate later task performance. Benchmark contamination and fine-tuning overlap further complicate generalization claims.
- Open-ended benchmarks: Open-ended benchmarks can mark semantically correct answers wrong when they differ from a specific ground-truth wording or expected response format.VQAv2 often expects answers of only one or two words, making template alignment influential.
- Open-ended benchmarks: 77.8 and 77.2 are the VQAv2 scores reported for Gemini 1.0 Ultra and GPT-4V, versus 82.0 for MM1-3B-Chat and 79.4 for moondream2.The smaller models had included some VQAv2 data in fine-tuning, complicating direct comparisons.
- Pre-training evaluation: Idefics2-base scores 57.9 on TextVQA with eight in-context examples and below 55 on DocVQA during pre-training, compared with 70.4 and 67.3 after fine-tuning in zero-shot evaluation.These open-ended tasks are described as less influenced by benchmark-specific templates.
- Pre-training evaluation: The pre-training-to-fine-tuning gap arises partly because visual question answering is learned during fine-tuning unless synthetic VQA is added as a third pre-training stage.Synthetic VQA examples are more aligned with benchmark tasks than captioning or transcription data alone.
- Pre-training evaluation: Without instruction data during pre-training, complex tasks may perform poorly and architecture effects may appear only after fine-tuning, making ablations misleading.The paper recommends including instruction data in the pre-training mixture for more accurate ablation insights.
- Contamination and overoptimization: MathVista contains at least 6.6% of questions with images from relevant training sets and 2.2% with identical or highly similar image-question pairs.At least 6.1% ask variants of an age-gap question also abundant in KVQA, favoring models exposed to similar fine-tuning examples.
- Contamination and overoptimization: Fine-tuning on benchmark-like examples can raise scores while providing little evidence of real-world generalization, so evaluated benchmark images should be excluded from SFT data.The paper frames benchmarks as measurement tools rather than training objectives.
5 Idefics3: adapting Llama 3 to multimodality
Idefics3 extends Idefics2’s data, architecture, and training pipeline with additional instruction datasets and Docmatix, a large-scale document-understanding dataset. Its pixel-shuffle and tiled-image design supports higher-resolution visual processing, while staged training and supervised fine-tuning yield strong task performance, especially on documents, though instruction following remains imperfect for challenging prompts.
- 5.1.1 Extending The Cauldron: The Cauldron is expanded with six datasets covering JSON output, visual question answering, detailed captioning, geometry, and document understanding.The additions are Cordv2, LNQA, ShareGPT-4o, IIW-400, Geo170K, and Docmatix.
- 5.1.2 Enhancing document understanding capabilities with Docmatix: 15% of generated Docmatix QA pairs are discarded after filtering for code and “unanswerable” answers.Phi-3-small generates QA pairs from English PDFA transcriptions using five prompts.
- 5.1.2 Enhancing document understanding capabilities with Docmatix: Training on a small Docmatix subset produces a nearly 20% relative improvement in Florence-2 document-understanding performance.The ablation uses 20% of Docmatix images and 4% of its QA pairs before an epoch on DocVQA.
- 5.2.1 Architecture and training methods: Idefics3 replaces Idefics2’s perceiver resampler with pixel shuffle and processes images as resolution-dependent 364x364 tiles plus a downscaled full image.Tile-position tokens and row separators preserve spatial organization in the visual sequence.
- 5.2.1 Architecture and training methods: Idefics3 uses three pre-training stages followed by supervised fine-tuning, progressively increasing resolution, adapting backbones with DoRA, and introducing large synthetic datasets.Fine-tuning applies NEFTune noise and computes loss only on answer tokens; the authors note that full unfreezing might improve performance.
- 5.2.2 Evaluation: Idefics3 improves over Idefics2-8B by 13.7 points on DocVQA and shows particularly strong gains on document-understanding tasks.The authors attribute gains to more visual tokens, a third pre-training stage with high-quality synthetic data, and an improved language backbone.
- 5.2.2 Evaluation: Idefics3 can extract CV information, recreate websites from screenshots with HTML, and summarize research papers from screenshots, but may struggle with difficult instructions.A brief assistant-response prefix can help users shape the generated output.
6 Conclusion
The paper combines a tutorial on VLM development with practical construction of Idefics3-8B and Docmatix, reporting improved document understanding and releasing the resulting resources.
- The tutorial analyzes architecture, data, and training choices across current VLM approaches, highlighting their strengths, weaknesses, and research directions.
- Idefics3-8B demonstrates significant improvements in document understanding, particularly through the introduced Docmatix dataset.
- The authors release both the Idefics3-8B model and the datasets created for its training.
A.1.1 Prompts used for the evaluation
Evaluation uses task-specific prompting: a default multiple-choice template for MMStar, MMMU, and MathVista, and Gemini prompts for TextVQA and DocVQA.
- MMStar is evaluated with a default multiple-choice template that presents the question and choices and requests the answer letter.
- MMMU and MathVista are evaluated with VLMEvalKit, using the default template for their multiple-choice questions.
- TextVQA and DocVQA use the prompts from Gemini for both evaluation and training.
TextVQA
The TextVQA prompt requires short answers grounded in the image, with explicit formatting rules for binary questions, times, punctuation, quotation marks, and common-sense reasoning.
- The prompt instructs the model to answer image questions using as few words as possible and to favor one- or two-word answers.
- Binary questions must receive Yes or No, while time questions must reproduce the time shown in the image.
- Answers should omit final full stops and quotation marks, avoid common-sense knowledge, and rely on information found in the image.
DocVQA
The DocVQA prompt requests terse answers that preserve text seen in the image without paraphrasing or reformatting, and generation stops at specified tokens.
- The DocVQA prompt requests a short, terse answer without paraphrasing or reformatting text visible in the image.
- Responses should contain no full stops and no additional explanation beyond the answer.
- Generation stops on Question, User, <end_of_utterance>, or the EOS token.
A.1.2 Detailed performance on MMMU
This section presents Idefics3’s detailed performance across the individual MMMU categories in Table 4.
- Table 4 reports Idefics3’s performance for each MMMU category.
- The results are organized as a category-level breakdown of MMMU performance.
- MMMU category performance is presented as a detailed evaluation of Idefics3.