PaperlayerView exact revision on arXiv

Source-linked AI summary

VILA: On Pre-training for Visual Language Models

Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, Song Han

arXiv:2312.07533v4cs.CV

TL;DR

Visual language pre-training has lacked an in-depth study of how design choices affect downstream performance. VILA tests those choices through controlled comparisons and outperforms the state-of-the-art model [38] across a wide range of vision-language tasks.

  • Problem

    The impact of visual language pre-training design choices on downstream performance has not been studied in depth.

    Source: §1 · p. 1

  • Method

    The authors use controlled comparisons in a pre-training-plus-SFT pipeline to test LLM updating, interleaved visual language data, and text-only instruction-data blending.

    Source: §1 · pp. 1–2, §3.3 · p. 5

  • Results

    VILA outperforms the state-of-the-art model [38] by noticeable margins across a wide range of vision-language tasks and shows multi-image reasoning, stronger in-context learning, and enhanced world knowledge.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that VILA’s recipe supports vision-task performance while preserving text-only capabilities.

    Source: §6 · p. 9

    Limited compute prevented scaling the pre-training corpus to billion-scale, which the authors leave for future work.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Visual language models (VLMs) rapidly progressed with the recent success of large language models. There have been growing efforts on visual instruction tuning to extend the LLM with visual inputs, but lacks an in-depth study of the visual language pre-training process, where the model learns to perform joint modeling on both modalities. In this work, we examine the design options for VLM pre-training by augmenting LLM towards VLM through step-by-step controllable comparisons. We introduce three main findings: (1) freezing LLMs during pre-training can achieve decent zero-shot performance, but lack in-context learning capability, which requires unfreezing the LLM; (2) interleaved pre-training data is beneficial whereas image-text pairs alone are not optimal; (3) re-blending text-only instruction data to image-text data during instruction fine-tuning not only remedies the degradation of text-only tasks, but also boosts VLM task accuracy. With an enhanced pre-training recipe we build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells and whistles. Multi-modal pre-training also helps unveil appealing properties of VILA, including multi-image reasoning, enhanced in-context learning, and better world knowledge.

1. Introduction

The introduction argues that visual-language pre-training is under-studied despite its importance for modality alignment, and examines how pre-training and instruction-tuning choices affect downstream capabilities. It presents VILA as a model built from these findings, with broad task gains and additional capabilities.

VILA results

  • VILA’s enhanced pre-training outperforms the state-of-the-art model [38] by noticeable margins across a wide range of vision-language tasks.Figure 1 compares VILA’s downstream accuracy with recent methods.Source: Figure 1 · p. 1, §1 · p. 2

Motivation

  • VLMs connect separately pre-trained LLMs and vision foundation models through joint training, yet prior work focused more on instruction tuning than on costly, critical pre-training.Augmenting LLMs with visual inputs can bring instruction following, zero-shot generalization, and few-shot in-context learning to visual-language tasks.Source: §1 · p. 1

Pre-training and SFT findings

  • Freezing the LLM yields decent zero-shot performance but not ICL; unfreezing supports ICL, interleaved data improves pre-training, and text-only SFT data remedies degradation while boosting visual accuracy.The authors attribute the ICL benefit of updating the LLM to deep embedding alignment.Source: §1 · pp. 1–2

VILA results

  • VILA’s pre-training is associated with multi-image reasoning, stronger in-context learning, and enhanced world knowledge.The paper reports multi-image reasoning even though the model sees only single image-text pairs during SFT.Source: §1 · p. 2

2. Background

The study focuses on autoregressive VLMs, which combine visual tokens with text for LLM input. It examines a three-stage process for adding visual support to a pre-trained text-only LLM and evaluates visual-language task performance, including in-context learning.

Model architecture

  • Autoregressive VLMs concatenate image and text tokens for LLM input and use a visual encoder, LLM, and projector to connect the modalities.The projector may be a linear layer [39] or use Transformer blocks [18].Source: §2 · p. 2

Training stages

  • Training proceeds from projector initialization with the ViT and LLM frozen, to visual-language pre-training, and finally visual instruction-tuning.Projector initialization uses image-caption pairs; pre-training considers interleaved image-text corpora and image-text pairs, while instruction tuning uses FLAN-style dataset-specific prompts.Source: §2 · p. 2

Evaluations

  • Ablations are evaluated on OKVQA and TextVQA accuracy and COCO and Flickr CIDEr, with both zero-shot and four-shot performance measured.The shot settings are used to reflect in-context learning capability.Source: §2 · p. 2

3. On Pre-training for Visual Language Models

The section finds that updating the LLM, pre-training on interleaved visual-text data, and blending text-only instruction data during fine-tuning help VLMs retain language abilities while improving visual-language performance. Its ablations link these choices to stronger in-context learning and reduced text-only degradation.

3.1. Updating LLM is Essential

  • Freezing the LLM preserves 0-shot performance but degrades 4-shot in-context learning, especially on out-of-distribution captioning tasks.The authors also find that training only the projector during SFT performs poorly, even with a high-capacity projector.Source: Table 1 · p. 3, §3.1 · p. 3
  • A simple linear projector slightly improves accuracy over a Transformer projector, which the authors hypothesize forces the LLM to learn more about visual inputs.Deeper visual-text embedding similarity rises alongside 4-shot accuracy, supporting a positive relationship between deep alignment and in-context learning.Source: §3.1 · p. 3, Figure 3 · p. 3

3.2. Interleaved Visual Language Corpus Helps Pre-training

  • MMC4 interleaving—not simply its longer text distribution—drives better visual-language accuracy and in-context learning than MMC4-pairs, while producing lower training loss.The table summary reports that blending interleaved and caption data further improves diversity and downstream accuracy.Source: Table 3 · p. 4, §3.2 · pp. 4–5, Figure 5 · p. 4
  • 17.2%: image-text-pair pre-training on COYO degrades MMLU text-only accuracy by this amount and yields worse visual-language accuracy than MMC4 pre-training.Its 4-shot accuracy is even worse than 0-shot, whereas MMC4 pre-training limits MMLU degradation to about 5% and yields higher 4-shot than 0-shot accuracy.Source: §3.2 · p. 4

3.3. Recover LLM Degradation with Joint SFT

  • Joint SFT with 1M FLAN text-only instructions restores text-only capability and improves visual-language performance.MMLU accuracy is on par with the original Llama-2 model fine-tuned on the same text-only instruction data; the authors hypothesize that better instruction following helps visual-language tasks.Source: Table 4 · p. 5, §3.3 · p. 5

3.2. Interleaved Visual Language Corpus Helps Pre-training

  • Blending MMC4 and COYO during pre-training prevents the severe text-only accuracy degradation from image-text pairs alone and further boosts visual-language benchmark accuracy.The authors attribute the benefit to greater corpus diversity.Source: §3.2 · p. 5

4. Experiments

The experiments show that VILA improves on LLaVA-1.5 across most visual-language benchmarks and exhibits capabilities including multi-image reasoning and in-context learning. They also examine how scaling, resolution, training choices, and interleaved data affect performance.

4.1. Scaling up VLM pre-training

  • VILA scales visual-language pre-training with 336×336 images, larger LLMs, diverse mixed pre-training data, and a higher-quality SFT blend.The corpus contains about 50M images, not billion-scale data; it mixes interleaved image-text data with image-text pairs at roughly 1:1 image proportions, and the SFT blend includes reference-based annotations.Source: §4.1 · p. 5

4.2. Quantitative Evaluation

  • VILA consistently improves over LLaVA-1.5 on most benchmarks in head-to-head comparisons, and its 7B model surpasses LLaVA-1.5 13B on VisWiz and TextVQA.VILA also outperforms LLaVA-1.5 on MMBench-Chinese despite using English visual-language instruction data.Source: Table 5 · p. 6, §4.2 · p. 6
  • VILA retains text-only performance close to a Llama-2 model fine-tuned on the same text SFT data: 7B is slightly lower, while 13B is higher.The comparison evaluates MMLU, BBH, and DROP; the authors suspect the smaller model undergoes greater text-performance degradation during pre-training.Source: §4.2 · p. 6, Table 6 · p. 6

4.3. Qualitative Evaluation

  • VILA reasons across multiple images despite single-image SFT, identifying shared objects and image differences in examples where LLaVA fails or hallucinates.The examples include recognizing a flamingo across three images and distinguishing headwear between two images.Source: §4.3 · p. 6, Figure 6 · p. 7
  • VILA outperforms LLaVA-1.5 on in-context image tasks, while LLaVA-1.5 fails examples involving OCR and repeating a prior example’s semantics.The authors note that LLaVA-1.5 can perform some in-context learning, which they believe is inherited from its text-only pre-training.Source: §4.3 · pp. 6–7, Figure 7 · p. 7
  • VILA performs visual chain-of-thought reasoning when prompted to “Think step-by-step” and correctly recognizes all four landmark samples, versus two for LLaVA-1.5.The authors believe the visual CoT capability is inherited from text-only SFT, without corresponding visual-language instruction examples.Source: §4.3 · pp. 7–8, Figure 8 · p. 7

4.4. Other Learnings.

  • TextVQA accuracy rose from 41.6% at 224 resolution to 49.8% at 336; with 336 resolution and token downsampling to 144 tokens/image, it was ~46%.Downsampling performed below the 336+linear setup by about 3%, but above the 224+linear setup; the authors report smaller gaps on OKVQA and COCO.Source: §4.4 · p. 8, Table 7 · p. 8
  • Directly fine-tuning the LLM during pre-training yields better VLM accuracy and in-context learning than adding a visual expert or using rank-64 LoRA.The visual expert nearly doubles model size; reformatting interleaved data as <im1><im2><txt1><txt2> instead of <im1><txt1><im2><txt2> degrades average 0-shot accuracy by 4.4% and 4-shot accuracy by 37.5%.Source: §4.4 · p. 8, Table 8 · p. 8, Table 9 · p. 8

5. Related Work

VLMs augment LLMs with visual inputs through cross-attention or autoregressive designs; this work studies autoregressive VLM pre-training. Prior work explores different training recipes, while this paper provides a holistic ablation of pre-training design choices.

Large language models (LLMs)

  • This work starts with the base Llama-2 model [61], against a backdrop of efforts to build strong base LLMs, conversational variants, and parameter-efficient fine-tuned versions.LLMs are based on Transformers [62], and their capabilities are attributed largely to scale in model size and pre-training corpus, with instruction tuning later unlocking capabilities [17] [47].Source: §5 · p. 8

Visual language models (VLMs)

  • Autoregressive VLMs tokenize visual inputs and feed them alongside text, while cross-attention designs fuse visual information into intermediate embeddings with the LLM frozen.The paper studies autoregressive VLM pre-training for its flexibility with multimodal inputs.Source: §5 · p. 9
  • Prior VLM pre-training varies whether the LLM is frozen or fine-tuned and uses corpora such as image-text pairs, interleaved datasets, video-text pairs, and visual-grounded annotations.This work presents a holistic ablation of these pre-training design choices.Source: §5 · p. 9

6. Conclusion

VILA’s pretraining recipe surpasses state-of-the-art vision methods while preserving text-only capabilities, and supports strong reasoning across multi-image analysis, in-context learning, and zero-/few-shot tasks.

6. Conclusion

  • VILA surpasses state-of-the-art methods for vision tasks while preserving text-only capabilities.The recipe leverages the full strength of LLM learning, interleaved image-text data, and careful text-data re-blending.Source: §6 · p. 9
  • VILA demonstrates strong reasoning capability for multi-image analysis, in-context learning, and zero-/few-shot tasks.Source: §6 · p. 9
  • The authors hope the work will spur further research on VLM pretraining and cross-modality dataset collection.Source: §6 · p. 9

A. SFT Blend for Ablation Study

The ablation SFT blend uses FLAN-style instructions built from 18 visual-language datasets following [18]; the final model also adds the LLaVA-1.5 SFT dataset [38].

A. SFT Blend for Ablation Study

  • The ablation blend uses FLAN-style instructions from 18 visual-language datasets, while the final model adds the higher-quality, more diverse LLaVA-1.5 SFT dataset [38].Most ablation datasets use a VQA format; the final blend includes visual-reference data such as RefCOCO [37] [44].Source: A. SFT Blend for Ablation Study · p. 13

B. Training Cost

Training the 7B model took 5.1k GPU hours, mostly in pre-training; the authors identify unimplemented throughput and image-resolution optimizations that could substantially reduce training time.

B. Training Cost

  • The 7B model used 16 A100 nodes (8 GPUs each) and 5.1k GPU hours total: 4 hours for projector initialization, 30 for pre-training, and 6 for instruction-tuning.Most computation was spent on pre-training.Source: B. Training Cost · p. 13
  • They did not use sample packing or length clustering, and estimate at least 30% less training time with optimization; 224×224 pre-training followed by later upscaling could reduce time by more than 50% [14].The reported 336×336 images corresponded to 576 tokens per image and lengthened training.Source: B. Training Cost · p. 13

C. Varying LLMs

With Vicuna-1.5-7B, the study reproduces its pre-training findings and reports similar benchmark accuracy across the two LLM backbones, concluding that its findings generalize across backbones.

C. Varying LLMs

  • The two LLM backbones achieved similar accuracy on the benchmarks, supporting the conclusion that the findings generalize across backbones.The comparison uses the setting of Table 5, reported in the right part of Table 11.Source: C. Varying LLMs · p. 13, Table 11 · p. 13
  • The recipe study with Vicuna-1.5-7B reached the same conclusion that updating the LLM is important during pre-training.The study reports average 0/4-shot accuracy in the left part of Table 11.Source: C. Varying LLMs · p. 13, Table 11 · p. 13
  • The authors tested whether their pre-training conclusions scale by using Vicuna-1.5-7B as a new LLM backbone and reproducing the recipe study.They also evaluated the setting of Table 5.Source: C. Varying LLMs · p. 13

D. Details on COYO Subsampling

To match the 25M images downloaded for MMC4-core, the authors subsampled COYO-700M by keeping the 25M samples with the highest image-caption CLIP similarity.

D. Details on COYO Subsampling

E. More Qualitative Samples

The qualitative samples illustrate VILA’s visual world knowledge, multimodal and visual reasoning, detailed captioning, and in-context learning capabilities. Several examples also contrast VILA’s responses with LLaVA-1.5 or BLIP-2.

Better world knowledge

More VQA samples

Visual reference understanding

More logical reasoning samples

Using VILA for detailed captioning

  • Prompted to describe images in detail, VILA produces more relevant captions than noisy LAION captions [54] and more detail than BLIP-2 [35].Examples include expanding a terse Sydney caption and describing details omitted by BLIP-2 in a surreal meat landscape.Source: E. More Qualitative Samples · p. 14, Figure 12 · p. 15

More in-context learning samples

Related papers

Loading 2312.07533v4…