PaperlayerView exact revision on arXiv

Source-linked AI summary

VILA: On Pre-training for Visual Language Models

Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, Song Han

arXiv:2312.07533v4cs.CV

TL;DR

The paper addresses the limited understanding of visual-language pre-training and examines how its design choices affect VLM performance. It compares training protocols and data mixtures, then builds VILA with an enhanced recipe that outperforms state-of-the-art models across vision-language tasks while retaining text-only capabilities.

  • Problem

    Visual-language pre-training lacks a thorough study of how its design choices affect downstream performance and the preservation of LLM capabilities.

  • Method

    The paper performs controlled ablations of LLM updating, corpus structure, and text-only data blending across visual-language pre-training and instruction tuning.

  • Results

    VILA outperforms state-of-the-art models across a wide range of vision-language tasks while preserving text-only capabilities.

  • Takeaways & Limitations

    The study provides practical guidance for VLM pre-training and reports multi-image reasoning, stronger in-context learning, and enhanced world knowledge in VILA.

  • Takeaways & Limitations

    The pre-training corpus was not scaled to billion-scale because of limited compute, although 50M images showed significant improvement.

Abstract

from arXiv · show

Visual language models (VLMs) rapidly progressed with the recent success of large language models. There have been growing efforts on visual instruction tuning to extend the LLM with visual inputs, but lacks an in-depth study of the visual language pre-training process, where the model learns to perform joint modeling on both modalities. In this work, we examine the design options for VLM pre-training by augmenting LLM towards VLM through step-by-step controllable comparisons. We introduce three main findings: (1) freezing LLMs during pre-training can achieve decent zero-shot performance, but lack in-context learning capability, which requires unfreezing the LLM; (2) interleaved pre-training data is beneficial whereas image-text pairs alone are not optimal; (3) re-blending text-only instruction data to image-text data during instruction fine-tuning not only remedies the degradation of text-only tasks, but also boosts VLM task accuracy. With an enhanced pre-training recipe we build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells and whistles. Multi-modal pre-training also helps unveil appealing properties of VILA, including multi-image reasoning, enhanced in-context learning, and better world knowledge.

1. Introduction

VILA studies how visual-language pre-training design choices affect downstream performance, addressing the limited prior study of this stage. Its recipe updates the LLM, uses interleaved data, and blends text-only instructions during SFT, producing broad task gains and additional capabilities.

  • Visual-language pre-training has received less thorough study than visual instruction tuning, despite its role in aligning vision and language at scale.
  • Updating the LLM during pre-training supports in-context learning, while freezing it provides decent zero-shot performance but lacks that capability.
  • Interleaved visual-language data is essential for pre-training, providing accurate gradient updates and maintaining text-only capability.
  • Blending text-only instruction data during SFT remedies text-only degradation and improves visual-language task accuracy.
  • VILA outperforms state-of-the-art models across a wide range of vision-language tasks and exhibits multi-image reasoning, stronger in-context learning, and enhanced world knowledge.

2. Background

The paper studies auto-regressive VLMs that connect a visual encoder and an LLM through a projector, with visual and text tokens generating text outputs. Its training pipeline initializes the projector, performs visual-language pre-training, then applies visual instruction tuning and evaluates zero- and four-shot performance.

  • Model architecture: Auto-regressive VLMs tokenize images into visual tokens concatenated with textual tokens and feed them into an LLM.
  • Model architecture: The architecture contains a visual encoder, an LLM, and a projector that bridges their embeddings.
  • Training stages: Projector initialization pre-trains the projector on image-caption pairs while freezing the separately pre-trained vision encoder and LLM.
  • Training stages: Visual-language pre-training updates the LLM and projector using either interleaved image-text corpora or image-text pairs.
  • Training stages: Visual instruction tuning follows pre-training, and ablations measure OKVQA and TextVQA accuracy plus COCO and Flickr CIDEr in zero- and four-shot settings.

3. On Pre-training for Visual Language Models

The study compares design choices for visual-language pre-training, focusing on LLM updating, corpus structure, and data blending. It finds that updating the LLM, using interleaved data, and blending text-only instruction data improve capability retention and downstream performance.

  • Updating LLM is Essential: Updating the LLM during pre-training preserves zero-shot performance while improving in-context learning compared with freezing it.Freezing the LLM degrades 4-shot performance, while deeper embedding alignment is positively related to in-context learning accuracy.
  • Updating LLM is Essential: Fine-tuning the LLM produces deeper visual-textual embedding alignment, which is associated with higher 4-shot accuracy.The comparison measures layerwise cosine similarity after excluding embedding-magnitude effects.
  • Interleaved Visual Language Corpus Helps Pre-training: Interleaved data substantially outperforms image-text pairs for preserving text-only capability and supporting visual-language in-context learning.MMLU degradation is 17.2% with COYO pairs but approximately 5% with MMC4 interleaved data; interleaved data also yields higher 4-shot than 0-shot accuracy.
  • Interleaved Visual Language Corpus Helps Pre-training: The benefit of interleaved data comes from its structure rather than merely its longer text distribution.MMC4-pairs removes interleaving while retaining images and corresponding text segments; MMC4 produces lower training loss because full text segments provide more information for language modeling.
  • Interleaved Visual Language Corpus Helps Pre-training: Blending interleaved and image-text data improves corpus diversity and downstream visual-language accuracy while preventing severe text-only degradation.Image-text-pair-only pre-training causes more than 17% text-only accuracy degradation, whereas blending reduces this degradation and further boosts visual-language benchmarks.
  • Recover LLM Degradation with Joint SFT: Adding text-only instruction data during SFT recovers text-only capability and improves visual-language accuracy despite its much smaller scale.Joint SFT blends 1M FLAN instructions and reaches text-only accuracy on par with the corresponding text-only-instruction-tuned Llama-2 model.

4. Experiments

Experiments show that VILA’s pre-training recipe and scaling choices improve visual-language performance while preserving competitive text-only ability. The model also exhibits stronger multi-image reasoning, in-context learning, and visual knowledge, with resolution and interleaving affecting accuracy.

  • Quantitative Evaluation: VILA achieves consistent improvements over LLaVA-1.5 across most visual-language datasets at different model sizes under head-to-head evaluation.The comparison uses the same prompts and base LLM; VILA 7B also outperforms LLaVA-1.5 13B on VisWiz and TextVQA.
  • Quantitative Evaluation: VILA maintains competitive text-only accuracy, with a small gap at 7B but higher accuracy than the text-only comparison at 13B.The evaluation covers MMLU, BBH, and DROP.
  • Qualitative Evaluation: VILA reasons over multiple images despite receiving only single-image samples during SFT, while LLaVA hallucinates or fails to separate information across images.Examples include identifying a shared flamingo across three images and distinguishing headwear between two images.
  • Qualitative Evaluation: VILA outperforms LLaVA-1.5 on in-context learning examples, including cases where LLaVA fails because of limited OCR or repeated semantics.The capability is evaluated with few-shot visual prompts.
  • Qualitative Evaluation: VILA correctly recognizes all four landmark locations in a case study, compared with two of four for LLaVA-1.5.The examples are used to illustrate improved visual-related world knowledge.
  • Other Learnings: Increasing resolution from 224 to 336 improves TextVQA accuracy from 41.6% to 49.8%, while downsampling to 144 visual tokens reaches approximately 46%.Higher resolution increases computational cost and limits the number of in-context demonstrations, whereas token compression retains some accuracy.
  • Scaling and Training Choices: Fine-tuning the LLM during visual-language pre-training outperforms freezing it or using LoRA for visual-language accuracy and in-context learning.Adding a visual expert nearly doubles model size, making it less suitable for edge deployment.
  • Other Learnings: Interleaved-data reformatting reduces average 0-shot accuracy by 4.4% and average 4-shot accuracy by 37.5%, showing the importance of data order for in-context learning.The reformatted MMC4 sequence places images before text rather than interleaving each image with its text.

5. Related Work

Related work extends language models with visual inputs through cross-attention or autoregressive architectures and explores varied training recipes and data sources. This paper focuses on pre-training design choices for autoregressive VLMs.

  • Visual Language Models: Visual language models augment LLMs with visual inputs through cross-attention-based or autoregressive designs.Autoregressive models tokenize images and feed visual and textual tokens jointly to the LLM.
  • Training Recipes and Data: Prior VLM training recipes either freeze the LLM while training auxiliary components or fine-tune the LLM to enable visual capabilities.Training corpora include image-text pairs, interleaved datasets, video-text pairs, and visual-grounded annotations.

6. Conclusion

VILA combines full LLM learning, interleaved image-text pre-training, and text-data re-blending to surpass state-of-the-art vision methods while preserving text-only capabilities.

  • VILA surpasses state-of-the-art methods for vision tasks while preserving text-only capabilities.Its recipe leverages full-strength LLM learning, interleaved image-text data, and careful text-data re-blending.
  • VILA demonstrates strong reasoning for multi-image analysis, in-context learning, and zero-shot and few-shot tasks.
  • The paper positions its pre-training findings and crossmodality data collection as directions for further research.

A. SFT Blend for Ablation Study

The ablation study uses an in-house FLAN-style instruction blend built from 18 visual language datasets, with an additional LLaVA-1.5 blend for the final model.

  • The ablation SFT blend derives FLAN-style instructions from the training sets of 18 visual language datasets.Most of these datasets use a visual question-answering format.
  • The final model additionally blends the LLaVA-1.5 SFT dataset for better quality and diversity.The cited example is visual reference data such as RefCOCO.

B. Training Cost

Training the 7B model requires 5.1k GPU hours, with most computation spent on visual language pre-training; throughput and image-resolution changes could substantially reduce this cost.

  • 5.1k GPU hours cover the 7B model’s three training stages across 16 A100 GPU nodes.Projector initialization takes 4 hours, visual language pre-training 30 hours, and visual instruction-tuning 6 hours per stage schedule.
  • Most computation is spent on the visual language pre-training stage.
  • At least 30% of training time could be reduced through throughput optimization, while lower-resolution pre-training may reduce it by more than 50%.The authors identify sample packing, sample length clustering, and later resolution scaling as future optimizations.

C. Varying LLMs

Experiments with Vicuna-1.5-7B reproduce the conclusion that updating the LLM during pre-training is important, while the two tested backbones achieve similar results in another setting.

  • Vicuna-1.5-7B reproduces the conclusion that updating the LLM is important during pre-training.The comparison reports average 0/4-shot accuracy in the ablation study.
  • The two tested LLM backbones achieve similar results in the additional evaluation setting.

D. Details on COYO Subsampling

The COYO-700M dataset was reduced to 25M images to match the available MMC4-core images. Samples were selected by image-caption CLIP similarity to retain stronger correspondence.

  • 25M images were subsampled from COYO-700M to maintain a similar dataset size to the 25M downloadable MMC4-core images.The authors downloaded 25M of 30M MMC4-core images.
  • Samples were sorted by CLIP similarity between images and captions, and the 25M highest-similarity samples were retained.
  • The selection criterion assumes that higher CLIP similarity usually indicates better image-caption correspondence.

E. More Qualitative Samples

The qualitative examples show VILA handling visual references, detailed visual reasoning, captioning, in-context learning, world knowledge, and multi-image or video-frame questions. Across comparisons, the examples highlight capabilities that distinguish VILA from LLaVA-1.5 and earlier captioning systems.

  • World knowledge: VILA correctly recognizes all 4 of 4 landmark samples, while LLaVA-1.5 recognizes 2 of 4 and shows bias toward common cities.
  • VQA capabilities: VILA handles memes, multi-image or video-frame reasoning, and corner cases in autonomous driving in the provided VQA samples.Examples include explaining a meme, inferring a lunch item from timed images, and recommending a safe response to a road hazard.
  • Visual reference understanding: VILA correctly identifies the contents of a circled glass in a visual-reference example where LLaVA-1.5 fails.The example distinguishes two bottles and links the circled glass to the water bottle.
  • Logical reasoning: VILA uses detailed visual features for logical reasoning, whereas LLaVA-1.5 does not produce reasonable responses in the cited samples.
  • Detailed captioning: VILA generates high-quality, detailed captions that can improve on noisy, irrelevant, or abbreviated dataset captions.Examples include a nighttime cityscape with boats and a surreal landscape made of meat, while BLIP-2 gives less accurate descriptions.
  • In-context learning: VILA demonstrates strong in-context learning across company knowledge, object counting, and French poems.
  • Visual reasoning: VILA explains visual physical relationships in a car comparison by connecting streamlined shape with reduced air resistance and drag.
Loading 2312.07533v4…