PaperlayerView exact revision on arXiv

Source-linked AI summary

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, Yinfei Yang

arXiv:2403.09611v4cs.CVcs.CLcs.LG

TL;DR

The paper asks which architectural and data choices matter for building performant MLLMs, addressing limited transparency around multimodal pre-training design. It uses comprehensive ablations and scales the resulting recipe into MM1 models up to 30B parameters. MM1 achieves strong few-shot and pre-training results, competitive post-SFT benchmark performance, and multi-image reasoning capabilities.

  • Problem

    Existing MLLMs disclose little about the data, architecture, and training decisions behind multimodal pre-training, limiting the available evidence for extracting reusable design lessons.

  • Method

    The paper performs ablations of visual encoders, vision-language connectors, and pre-training data mixtures, then scales the recipe across dense and MoE MM1 models.

  • Results

    MM1 is SOTA on pre-training evaluations and achieves competitive performance after SFT across established multimodal benchmarks, while supporting few-shot and multi-image reasoning.

  • Takeaways & Limitations

    Image resolution and visual encoder choices matter substantially, connector architecture matters comparatively little, and data type importance depends on the evaluation setting.

  • Takeaways & Limitations

    The AIM comparison is inconclusive because AIM was trained on less than half the data, while OpenAI's model performs on-par at comparable capacity with a different data mixture.

Abstract

from arXiv · show

In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models up to 30B parameters, including both dense models and mixture-of-experts (MoE) variants, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.

1 Introduction

MM1 documents how multimodal pre-training design choices affect model performance and scales the resulting recipe into competitive models with in-context and multi-image reasoning abilities.

  • The paper addresses limited transparency about multimodal pre-training decisions by documenting the MLLM building process and extracting design lessons.
  • Ablations identify image resolution, visual encoder loss and capacity, and visual encoder pre-training data as more important than how visual data enters the LLM.
  • Interleaved and text-only data are most important for few-shot and text-only performance, whereas caption data matters most for zero-shot performance.
  • Scaling from 3B to 30B parameters, including MoE variants, produces models that outperform relevant systems on few-shot captioning and VQA and remain competitive after SFT.
  • Large-scale multimodal pre-training supports in-context predictions, multi-image reasoning, chain-of-thought reasoning, and strong few-shot learning after instruction tuning.

2 Related Work

Related work has established decoder-only multimodal models and extensive instruction-tuning research, but thorough pre-training ablations and optimization disclosures remain relatively sparse.

  • The studied MLLMs build on pre-trained autoregressive LLMs that consume text and visual tokens produced by an image encoder, using a decoder-only architecture.
  • Prior work extensively studies visual instruction tuning, instruction-tuning data, referring and grounding, and image generation or editing.
  • Thorough pre-training ablations are relatively sparse: VILA lacks optimization details and detailed evaluations, while Emu2 lacks ablations justifying component decisions.
  • MM1 provides detailed pre-training components, base-model results, and extensive ablations of visual encoders, connectors, and data mixtures.

3 Recipe for Building MM1

MM1’s recipe is developed through controlled ablations of architecture, data, and training choices, then scaled to larger models. The experiments identify image resolution, encoder properties, visual-token count, and data mixtures as important design factors, while connector architecture has comparatively little effect.

  • Empirical setup: The ablation recipe varies image encoders, visual-language connectors, data types and mixture weights, and training-procedure choices using a smaller model before scaling.Components are modified one at a time to assess their impact efficiently.
  • Empirical setup: A 1.2B decoder-only language model is paired with a ViT-L/14 image encoder, C-Abstractor connector, 144 image tokens, and a 45:45:10 captioned/interleaved/text-only data mix.The base image resolution is 336×336.
  • Image encoder ablations: Image resolution has the largest encoder impact: increasing it from 224 to 336 produces an approximate 3% boost across metrics, whereas doubling encoder size usually adds less than 1%.Adding VeCap-300M yields more than a 1% few-shot boost.
  • Image encoder ablations: ViT-L encoders outperform comparable AIM600M encoders by 0.3% to 1.5%, but the comparison is inconclusive because AIM used less than half the training data.The open-source OpenAI model performs on par with a comparable-capacity DFN+VeCap model.
  • Vision-language connector ablations: Increasing visual-token count or image resolution improves zero- and few-shot performance, while different connector architectures do not conclusively produce stronger models.After instruction tuning, the three architectures achieve very similar results at 336px and 144 tokens.
  • Pre-training data ablations: Interleaved data is crucial for few-shot and text-only performance, caption data lifts zero-shot performance, and a 5:5:1 caption/interleaved/text ratio balances multimodal and text understanding.VeCap adds 2.4% and 4% absolute few-shot gains.

4 Final Model and Training Recipe

MM1’s final recipe combines a high-capacity vision encoder, many visual tokens, a balanced multimodal data mixture, and scaled language models. The resulting pretrained models achieve strong few-shot performance, while learning-rate selection is extrapolated from smaller-scale experiments.

  • Final recipe: The final recipe uses a ViT-H encoder at 378×378 resolution, a 144-token C-Abstractor connector, and a 45% interleaved, 45% image-text pair, 10% text-only mixture.The connector architecture is selected because visual-token count matters more than connector type.
  • Scaling: MM1 scales the LLM backbone to 3B, 7B, and 30B parameters while pre-training for 200k steps, approximately 400B tokens, with up to 16 images per sequence.The models are trained with sequence length 4096 and batch size 512 sequences.
  • Training procedure: Learning-rate scaling uses small-model grid searches and log-space linear regression to extrapolate the optimal peak learning rate to larger models.For 3e10 nonembedding parameters, the fit predicts η = 2.2e−5, used for MM1-30B.
  • Pre-training results: The pretrained MM1 models outperform published prior work on few-shot evaluations, with especially strong 30B results across captioning and VizWiz-QA.On VQAv2, TextVQA, and OKVQA at that scale, performance is comparable to Emu2.
  • Evaluation: Zero-shot results are reported for reference, while model selection intentionally does not optimize those metrics because they mainly reflect task-format alignment.The authors emphasize few-shot pre-training performance as the primary comparison.

5 Supervised Fine-Tuning

Supervised fine-tuning uses roughly 1.45M examples spanning generated conversations, academic vision-language tasks, and text-only instruction following. Higher-resolution inputs and continued pre-training improve results, while the resulting models perform strongly across benchmarks and support few-shot, multi-image reasoning.

  • SFT data mixture: The SFT mixture contains roughly 1.45M examples from generated instruction data, academic vision-language datasets, and text-only instruction-following data.The data covers conversations, detailed image descriptions, natural-image VQA, OCR, documents, charts, and text-only instruction following.
  • SFT procedure: The models are evaluated across 12 benchmarks, with both the image encoder and LLM backbone kept unfrozen during SFT.Academic datasets are formatted as instruction-following examples and randomly sampled together during training.
  • Benchmark results: MM1-3B-Chat and MM1-7B-Chat outperform all listed same-size models on average, with particularly strong VQAv2, TextVQA, ScienceQA, MMMU, and MathVista results.The 30B chat model also outperforms Emu2-Chat-37B and CogVLM-30B on TextVQA, SEED, and MMMU.
  • Resolution: A 1344×1344 image resolution yields a 15% relative increase over 336 pixels in average SFT performance, while 1792×1792 decreases performance slightly.The standard Table 4 results use 1344×1344 resolution.
  • Pre-training impact: SFT performance consistently improves with more pre-training data, and large-scale multimodal pre-training enables in-context few-shot learning and multi-image reasoning.These capabilities extend beyond the zero-shot and single-image focus of most listed benchmarks.
  • Few-shot reasoning: Four-shot MathVista performance reaches 41.9 versus 39.4 zero-shot, and mixed-resolution eight-shot prompting reaches 44.4.The mixed-resolution approach allows more in-context examples despite using 720 visual tokens per image at high resolution.
  • Transfer of lessons: Pre-training lessons transfer to SFT: caption-only pre-training improves SFT metrics, while different vision-language connector architectures have negligible final impact.Detailed ablations are provided in Appendix C.4.

6 Conclusion

The paper studies how modeling and data choices affect performant MLLMs through systematic ablations. Its recipe produces competitive models with few-shot prompting and multi-image reasoning, while the interleaved-data corpus is constructed from large-scale filtered web documents.

  • 6 Conclusion: The study ablates modeling and data choices to identify design lessons for building performant MLLMs beyond a single architecture or data strategy.The authors report that these lessons yield strong pre-training results and competitive post-SFT performance.
  • 6 Conclusion: The final model family achieves competitive performance across many benchmarks while enabling multi-image reasoning and few-shot prompting.The conclusion presents these capabilities as outcomes of the presented recipe at scale.
  • Interleaved data: The interleaved corpus contains 500M image-text documents, 1B images, and 500B text tokens, built from 3B HTML files.Documents are constructed by extracting text and image tags, retaining images at their original positions.
  • Interleaved data: Image filtering removes corrupted, extreme-aspect-ratio, improperly sized, and likely UI-related images before de-duplication.Repeated URLs or MD5 hashes and repeated images within a page are also filtered.

A.2 Text-Only Data

The text-only and supervised fine-tuning data are assembled from filtered web content, generated instruction examples, academic vision-language datasets, and a small text-only component. These sources target conversation, vision-language understanding, and text-only instruction following.

  • Text-only data: An initial corpus of 150B English HTML files is filtered and de-duplicated to produce 3B HTML files for downstream data construction.Processing removes boilerplate, short or profane documents, low-quality content, and exact or near duplicates.
  • SFT data: The SFT mixture includes GPT-4 and GPT-4V-generated conversations and detailed image descriptions.The cited sources include LLaVA-Conv, LLaVA-Complex, and ShareGPT-4V.
  • SFT data: Academic vision-language data covers natural-image captioning and VQA, text-rich images, and document and chart understanding.Examples include VQAv2, GQA, TextCaps, DocVQA, ChartQA, and InfoVQA.
  • SFT data: A small amount of text-only SFT data is blended in to improve text-only instruction following capability.The text-only component is included alongside the multimodal instruction data.
  • Data formatting: Academic vision-language datasets are converted to instruction-following format, and all datasets are mixed and randomly sampled during training.The formatting follows LLaVA-1.5.

B.1 Pre-training

MM1 pre-training uses a fixed, reproducible mixture of multimodal and text-only inputs, with model-size-specific optimization settings and standardized evaluation procedures.

  • Pre-training setup: MM1 pre-training samples interleaved, packed image-text, and text-only inputs using batches of 512 and sequences up to 4096 tokens.Each sequence may contain up to 16 images, each represented by 144 decoder input tokens.
  • Pre-training setup: The sampling procedure is stored as a deterministic offline snapshot, keeping examples and ordering consistent across models except mixture ablations.This was intended to improve internal reproducibility after different input-pipeline seeds produced non-negligible result changes.
  • Optimization: MM1 uses cosine learning-rate decay with 2000 warmup steps, decaying to 10% of peak over 2e5 training steps.Training uses gradient clipping, AdamW, and a decoupled weight-decay implementation; MM1-30B additionally uses a z-loss term with scale 1e-4.
  • Optimization: The actual MM1 peak learning rates are 6e-5, 4e-5, and 2e-5 for the 3B, 7B, and 30B models, respectively.The vision-language connector uses η = 8e-5 for all model sizes.
  • SFT setup: SFT runs for 10k steps with batch size 256, sequence length 2048, AdaFactor, peak learning rate 1e-5, and cosine decay to 0.Both the image encoder and LLM remain unfrozen during SFT because full-model fine-tuning performed better empirically.

C.2 SFT Evaluation Benchmarks

MM1 SFT models are evaluated on established academic vision-language benchmarks and newer multimodal-LLM benchmarks, using the specified pre-training evaluation splits.

  • Evaluation splits: Pre-training ablations use validation splits for VQAv2 and VizWiz, unlike the main pre-training results.The split distinction is explicitly noted for pre-training evaluation.
  • Benchmark coverage: The SFT evaluation covers VQAv2, TextVQA, image ScienceQA, POPE, MME, MMBench, SEED-Bench, LLaVA-Bench-in-the-Wild, MM-Vet, MathVista, and MMMU.These include both traditional academic VL tasks and benchmarks designed specifically for MLLMs.
  • Evaluation procedure: Responses for all benchmarks are generated with greedy decoding.The benchmark passage specifies greedy decoding as the generation procedure for SFT evaluation.

C.3 SFT Evaluation Meta-Average

SFT ablation results are compressed into a normalized meta-average so metrics from different datasets can be compared against a common baseline.

  • Meta-average construction: Each task metric is divided by its respective baseline before averaging across metrics.This normalization addresses differences in metric ranges across datasets.
  • Ablation comparisons: Figure 10 compares SFT results for pre-training data mixtures, vision-language connectors, and image-encoder freezing choices.The compared models are first pre-trained under each ablated setting and then subjected to SFT.
  • Meta-average construction: The baseline is a compact MM1 model trained at 224×224 resolution with attention pooling and 64 image queries.The baseline configuration anchors the normalized comparison across SFT ablations.

C.4 Additional SFT Ablations

Additional SFT ablations examine connector choice and whether to freeze the image encoder, finding small connector effects and resolution-dependent encoder effects.

  • Vision-language connector: Vision-language connector choice has little impact on final test performance, and the final models use the C-Abstractor architecture.This conclusion is drawn from the SFT connector ablation.
  • Image encoder: +2.2 points favor freezing the image encoder at lower image resolutions during SFT.The comparison is between frozen and unfrozen image encoders.
  • Image encoder: +2.9 points favor unfreezing the image encoder at the higher 1344px resolution.The authors relate this pattern to pre-training at the base resolution without interpolation or image subdivisions.

C.5 Implementation Details for Few-shot MM1-30B-Chat

MM1-30B-Chat improves with more in-context examples, while mixed-resolution inputs reduce the token cost of sub-image decomposition and enable stronger few-shot chain-of-thought performance. Qualitative examples examine a broad range of multimodal capabilities, and the paper credits specific contributors for implementation, experimentation, and infrastructure.

  • Few-shot performance: Increasing the number of in-context examples raises MM1-30B-Chat performance.The fine-tuned model uses in-context examples to achieve stronger results, with performance increasing as examples are added.
  • Few-shot efficiency: Sub-image decomposition improves zero-shot performance but substantially increases the effective image-token budget for few-shot inputs.With 5 sub-images per image and 144 tokens per image, a 4-shot input containing one source image per example requires 2,880 image tokens.
  • Few-shot performance: 39.4 to 41.9: MathVista performance improves from 0-shot to 4-shot chain-of-thought with full sub-image decomposition.The 4-shot result uses in-context examples in which every example applies full sub-image decomposition.
  • Few-shot efficiency: 44.4: mixed-resolution inputs enable up to 8-shot chain-of-thought on MathVista, exceeding the 4-shot result.The mixed-resolution approach encodes additional in-context examples and improves performance from 41.9 to 44.4.
  • Qualitative capabilities: Qualitative evaluations cover counting, OCR, scientific knowledge, adversarial prompts, image alignment, aesthetics, multi-image comparison, navigation, graphics, and cross-image-text reasoning.Additional examples test instruction following across multiple images and applying a style or task across few shots; graphics examples include a highlighted confusion case.
  • Implementation details: The implementation involved dedicated leads for multimodal pre-training, supervised fine-tuning, evaluation infrastructure, image encoders, VL connectors, packed image-text data, interleaved documents, and MoE models.The contribution statements also identify roles in dataset creation, filtering, data processing, model implementation, and underlying LLM training.
Loading 2403.09611v4…