PaperlayerView exact revision on arXiv

Source-linked AI summary

xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant Kendre, Jieyu Zhang, Shaoyen Tseng, Gustavo A Lujan-Moreno, Matthew L Olson, Musashi Hinck, David Cobbley, Vasudev Lal, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin Choi, Ludwig Schmidt, Zeyuan Chen, Silvio Savarese, Juan Carlos Niebles, Caiming Xiong, Ran Xu

arXiv:2408.08872v4cs.CVcs.AIcs.CL

TL;DR

Open LMM research lacks accessible base models, training recipes, and curated datasets, while BLIP-2 is described as constrained by data scale, quality, diversity, and single-image input. BLIP-3 combines curated multimodal datasets, a scalable vision token sampler, and simplified training, and reports competitive results across multimodal benchmarks. The authors release models, datasets, and code to support broader community access and research.

  • Problem

    Open LMM research has limited access to strong base models, training recipes, and curated datasets, and BLIP-2 is described as limited in data scale, quality, diversity, and input to single images.

    Source: §1 · p. 1

  • Method

    BLIP-3 scales LMM training with interleaved and curated caption data, three specialized OCR and grounding datasets, a vision token sampler, and auto-regressive text-token training.

    Source: §1 · pp. 1–2

  • Results

    BLIP-3 LMMs achieve competitive results on a range of multimodal benchmarks.

    Source: §8 · p. 8

  • Takeaways & Limitations

    The authors release BLIP-3 4B and 14B models, curated datasets, and training code to make multimodal foundation resources more accessible to the research community.

    Source: §1 · p. 2, §8 · p. 8

Abstract

from arXiv · show

This paper introduces BLIP-3, an open framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. We release 4B and 14B models, including both the pre-trained base model and the instruction fine-tuned ones. Our models undergo rigorous evaluation across a range of tasks, including both single and multi-image benchmarks. Our models demonstrate competitive performance among open-source LMMs with similar model sizes. Our resulting LMMs demonstrate competitive performance among open-source LMMs with similar model sizes, with the ability to comprehend interleaved image-text inputs. Our training code, models, and all datasets used in this work, including the three largescale datasets we create and the preprocessed ones, will be open-sourced to better support the research community.

1. Introduction

The introduction identifies gaps in open LMM access, training transparency, and dataset diversity, then presents BLIP-3 as a framework built around broader data, a scalable vision token sampler, and simplified training.

1. Introduction

  • Open LMM research faces limited access to strong base models, training recipes, and curated datasets, while some strong open-weight models omit training details and full data recipes.The introduction links this lack of transparency to difficulties replicating, analyzing, and improving LMMs.Source: §1 · p. 1
  • BLIP-3 scales training with interleaved and curated caption datasets, replaces BLIP-2’s Q-Former with a vision token sampler, and uses text-token auto-regressive loss.The framework also unifies training objectives to a single loss at every training stage.Source: §1 · p. 1, Figure 1 · p. 2
  • BLIP-3 adds BLIP3-OCR-200M for base-resolution OCR, BLIP3-GROUNDING-50M for visual grounding, and BLIP3-OCR-HD-30M for high-resolution OCR pre-training.Source: §1 · p. 2
  • The release plan includes pre-trained base models, instruction-tuned models, and training code, alongside the datasets.The authors aim to make LMM research and development more accessible to the community.Source: §1 · p. 2

2. Related Work

Related work distinguishes LMM connector architectures and training strategies; BLIP-3 adopts self-attention and presents an open family of models and resources for future research.

2. Related Work

  • Self-attention connectors concatenate visual and textual tokens in one LLM input sequence, and BLIP-3 adopts this design for its simplicity.The approach is described as prevalent and effective across a range of LMMs [1] [2] [8] [76].Source: §2 · p. 2
  • LMM training commonly pairs light pre-training with extensive instruction tuning, as in LLaVA, or large-scale diverse pre-training followed by instruction fine-tuning, as in MM1 [60] [82] and Idefics2.MM1 offers extensive recipe ablations but releases limited resources for reproduction.Source: §2 · pp. 2–3
  • BLIP-3’s stated distinction is an open LMM family combining models, data recipes, training code, and three new large-scale foundational multimodal datasets.The paper frames these resources as supporting future community research.Source: §2 · p. 3

3. Model Architecture

BLIP-3 combines a vision transformer, a perceiver-resampler vision token sampler, and a pre-trained LLM to process interleaved text and image inputs, including high-resolution images.

3. Model Architecture

  • The architecture uses a ViT, a perceiver resampler to downsample image embeddings, and a pre-trained Phi-3-series LLM for free-form interleaved multimodal inputs.Source: §3 · p. 3
  • Any-resolution encoding splits an image into patches and combines their encodings with a downsized original image that retains global information.This strategy is used during Stage-2 pre-training and fine-tuning to support higher-resolution image understanding.Source: §3 · p. 3
  • The connector downsamples each image patch independently, concatenates the resulting vision tokens, and sends them to the LLM, reducing token sequence length by a factor of five or more depending on query-token count.Source: §3 · p. 3

4. Training

Training proceeds through base-resolution pre-training, high-resolution pre-training, single-image instruction fine-tuning, and a second fine-tuning stage for interleaved multi-image inputs.

4. Training

  • In this stage, the base model is trained on approximately 100 billion multimodal tokens from an ensembled mixture of open-source datasets and BLIP3-OCR-200M and BLIP3-GROUNDING-50M, at 384×384-pixel pre-training resolution for computational efficiency.Source: §4 · p. 3
  • Stage 2 continues from the Stage-1 checkpoint with additional high-quality, high-resolution data, including BLIP3-OCR-HD-30M, and supports up to 12 patches with a maximum longer-side length of 1536 pixels.It uses the same any-resolution vision token sampling strategy as supervised fine-tuning.Source: §4 · p. 3
  • Single-image supervised fine-tuning uses publicly available instruction-following datasets and the Stage-2 any-resolution sampling strategy for high-resolution images, including text-rich documents.Source: §4 · p. 3
  • A second fine-tuning stage mixes multi-image and single-image instruction-following samples to improve comprehension of interleaved image-text inputs for multimodal in-context learning and multi-image question answering.This stage retains the any-resolution sampling strategy.Source: §4 · p. 3

5. Data

BLIP-3 combines diverse Stage-1 pre-training data with specialized Stage-2 data and interleaved datasets. Its recipe also includes single-image and multi-image instruction-tuning mixtures.

5.1. Pre-training Data Recipe

  • Stage-1 pre-training uses a diverse multimodal dataset mixture with specified sampling ratios.The supplied passage refers to the recipe overview in Figure 3.Source: §5.1 · p. 3
  • Stage 2 replaces OCR-200M with OCR-HD-30M and adds IDL at 10% each, using images over 512 pixels per side for high-resolution OCR annotations.The OCR-HD-30M process yields 30 million samples; the remaining caption-dataset sampling ratios are adjusted.Source: §5.1 · pp. 3–4
  • The interleaved mixture combines MINT-1T HTML, PDF, and ArXiv subsets with HTML-only OBELICS; MINT-1T subsets are mixed at 7:5:1.OBELICS differs slightly in domain coverage because of its preprocessing.Source: §5.1 · p. 4
  • The caption mixture uses BLIP3-KALE as a detailed-caption base and adds OCR-200M, whose captions include OCR text with optional bounding-box information.BLIP-3 pre-training uses OCR text without the bounding-box data.Source: §5.1 · p. 4
  • BLIP3-GROUNDING-50M adds object locations to captions in three localization formats, using detections and tags for 50 million Datacomp-1B images.The dataset is intended to support grounding semantic concepts in visual features.Source: §5.1 · p. 4

5.2. Supervised Fine-tuning Data Recipe

  • Single-image visual instruction tuning uses 3 million public samples for one epoch, while multi-image tuning mixes interleaved data with some prior single-image data.The retained single-image samples are included to prevent deterioration of single-image capabilities.Source: §5.2 · p. 4

6. Experiments

The experiments evaluate BLIP-3 across single-image and multi-image multimodal benchmarks. Interleaved multi-image fine-tuning yields significant improvements on multi-image benchmarks.

Single-Image Evaluation

  • Evaluation spans general VQA, domain knowledge, OCR, hallucination, and multi-image tasks; Table 1 reports individual single-image scores and an overall benchmark average.Table 1 also includes GPT-4V as a reference and uses third-party scores or evaluation-codebase runs for some open-source models.Source: §6 · p. 4, Table 1 · p. 5

Experiments

  • Figures 3 and 4 show the Stage-1 dataset recipe and samples from OCR-200M, OCR-HD-30M, and the grounding dataset.The OCR-200M examples show six granularities with and without boxes; the OCR-HD-30M examples show Level 1 only.Source: Figure 3 · p. 5, Figure 4 · p. 6

Multi-Image Evaluation

  • Interleaved multi-image fine-tuning produces significant improvements on multi-image benchmarks compared with the single-image model.The paper notes that single-image supervised fine-tuning may degrade multi-image ability.Source: Table 2 · p. 6, §6 · p. 6

7. Ablation Studies

The ablations examine how pretraining scale, resolution, and vision-token sampling affect BLIP-3. Scaling data and adding Stage-2 pretraining improve few-shot or OCR performance, while the instruction-tuning studies compare sampling choices and find little difference from instruction-aware sampling.

7.1. Ablation on Stage-1 Pre-training

  • Scaling Stage-1 pretraining from 2B to 60B multimodal tokens substantially improves few-shot image-text and OCR performance, with moderate additional benefit at 100B.The evaluation uses few-shot CIDEr for captioning and accuracy for VQA; Figure 5 shows performance across pretraining-data scales.Source: §7.1 · pp. 6–7, Figure 5 · p. 7

7.2. Ablation on Stage-2 Pre-training

  • Adding Stage-2 pretraining significantly improves OCR while general VQA remains stable.The comparison uses a 100B-token Stage-1 model as baseline; models share a fine-tuning protocol and a maximum of 13 image patches.Source: §7.2 · p. 7, Table 4 · p. 7

7.3. Ablation on Instruction Tuning

  • The perceiver resampler scores lower on OCR but better in other domains and overall than the MLP projector, supporting its selection for efficiency with interleaved multi-image input.The two sampler designs use similar vision-compression ratios; the ablation results are from single-image fine-tuned checkpoints and are not directly comparable to the main results.Source: §7.3 · p. 7, Table 5 · p. 7
  • On text-rich OCR benchmarks, any-resolution encoding significantly improves results with downsampled vision tokens, and patch-wise sampling outperforms fixed sampling.Fixed sampling improves over the base-resolution baseline, but the authors suspect its high compression or concatenated embeddings may explain its weaker performance.Source: §7.3 · p. 8, Figure 7 · p. 8
  • The instruction-aware resampler shows no significant difference from the original on common task domains, so the authors retain the simpler original architecture.The authors leave further exploration of the instruction-aware connector to future work.Source: §7.3 · p. 8, Figure 7 · p. 8

8. Conclusion

BLIP-3 is a framework for training open-source multimodal models on curated large-scale datasets. Its 4B and 14B models achieve competitive benchmark results, and the authors release the models, datasets, and training code.

  • BLIP-3 provides an open framework for training multimodal models on a curated mixture of large-scale datasets.The released model sizes are 4B and 14B; the release also includes curated datasets and training code.Source: §8 · p. 8

Related papers

Loading 2408.08872v4…