PaperlayerView exact revision on arXiv

Source-linked AI summary

VILA$^2$: VILA Augmented VILA

Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jan Kautz, Jang Hyun Cho, Marco Pavone, Song Han, Hongxu Yin

arXiv:2407.17453v2cs.CV

TL;DR

VLM training data often consists of coarse captions, while large-scale human annotation is costly and distillation from proprietary models is bounded. VILA2 iteratively recaptions pretraining data and then adds specialist-generated task data. The authors report state-of-the-art performance across a comprehensive set of benchmarks.

  • Problem

    VLM training commonly uses coarse-quality caption pairs, while scaling human annotation is costly and proprietary-model distillation is bounded by those models’ performance.

    Source: §1 · pp. 1–2

  • Method

    VILA2 iteratively improves pretraining data through general self-augmentation, then uses SFT-finetuned spatial, grounding, and OCR specialists to generate task-aware data.

    Source: §1 · p. 2

  • Results

    VILA2 models demonstrate state-of-the-art performance across a comprehensive set of benchmarks.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The authors conclude that self-bootstrapping can provide three rounds of improvement, with further gains from specialist knowledge distillation.

    Source: §5 · p. 11

    The study verifies 7B, 8B, and 40B models using 51M data; larger models and more than 0.5B pretraining examples remain future work.

    Source: §4 · p. 11

Abstract

from arXiv · show

While visual language model architectures and training infrastructures advance rapidly, data curation remains under-explored where quantity and quality become a bottleneck. Existing work either crawls extra Internet data with a loose guarantee of quality or distills from black-box proprietary models, e.g., GPT-4V / Gemini that are API frequency and performance bounded. This work enables a VLM to improve itself via data enhancement, exploiting its generative nature. We introduce a simple yet effective VLM augmentation scheme that includes a self-augment step and a specialist-augment step to iteratively improve data quality and hence, model performance. In the self-augment step, the instruction-finetuned VLM recaptions its pretraining caption datasets and then retrains from scratch leveraging refined data. Without any expensive human-in-the-loop annotation, we observe improvements in data quality and downstream accuracy boosts with three self-augmentation rounds -- a viable free lunch to the current VLM training recipe. When self-augmentation saturates, we augment the caption diversity by leveraging specialty skills picked up from instruction finetuning. We finetune VLM specialists from the self-augmented VLM with domain-specific experts, including spatial, grounding, and OCR, to fuse task-aware synthetic data into the pretraining stage. Data quality improvements and hallucination reductions are cross-checked by VLM (GPT-4V, Gemini) and human judges. Combining self-augmentation and specialist-augmented training, VILA$^2$ consistently improves the accuracy on a wide range of benchmarks over the prior art, producing a reusable pretraining dataset that is 300x more cost-efficient than human labeling.

1 INTRODUCTION

VLM training has advanced faster than its data curation, while large-scale human annotation is costly and proprietary-model distillation is bounded. This work uses a VLM to improve its own pretraining data through self-augmentation and specialist augmentation, reporting performance gains across benchmarks.

1 INTRODUCTION

  • VLM training data remains largely coarse caption pairs followed by small-scale SFT, while scaling human annotation to millions or billions of examples is unrealistic.Distillation from proprietary models such as GPT-4V and Gemini is also bounded by those models’ performance.Source: §1 · pp. 1–2
  • VILA2 uses a self-augment loop to improve pretraining captions, then adds specialist-generated data after self-augmentation begins to saturate.Specialists focus on spatial, OCR, or grounding skills acquired through additional SFT data.Source: §1 · p. 2
  • VILA2 reports state-of-the-art performance across main benchmarks, with improved data quality and reduced hallucinations.The authors report that data-quality improvements were cross-checked by GPT-4V, Gemini, and human judges.Source: §1 · p. 2

2 METHODOLOGY

VILA2 first bootstraps better general captions, then uses task specialists to add spatial, grounding, and OCR information to pretraining data. The method relies on detailed prompts and retaining original human captions alongside synthetic recaptions.

2 METHODOLOGY

  • VLMs project image tokens into textual space and use a three-stage align-pretrain-SFT training paradigm in this study.The method first bootstraps general captions, then uses SFT-acquired skills as specialist feedback for pretraining.Source: §2 · p. 2

2.1 SELF-AUGMENTING VIA GENERAL KNOWLEDGE ENHANCEMENT

  • VILA0 recaptions pretraining images with detailed captions, and the next VILA round trains on those synthetic texts paired with the original images.The loop can be repeated to convergence, with reported gains in VQAv2 and LLM-judged text quality.Source: §2.1 · p. 3

2.1.1 PROMPTS AND TEMPLATE DESIGN

  • Brief recaptioning produced longer captions but no notable benchmark improvement, whereas detailed prompts improved recaption quality and benchmark performance.The authors use a random mixture of three long prompt versions; they also report that concatenating original and recaptioned texts consistently improves all VLM metrics.Source: §2.1.1 · pp. 3–4

2.2 SURPASSING THE LIMIT WITH SPECIALIST VLM AUGMENTATION

  • Because general self-augmentation begins to saturate, VILA2 adds task-specific knowledge through specialist VLMs finetuned for spatial relations, localization, and OCR.Specialist annotations are appended to original captions as QA pairs for the next pretraining iteration.Source: §2.2 · p. 4, §2.2.1 · p. 5

2.2.1 ACQUIRING SPECIALIZED KNOWLEDGE

  • The specialist stage covers spatial relations, grounded narration, and OCR, using task-oriented prompts to generate annotations for pretraining images.SpatialRelationQA is built from 3D scenes; grounding data comes from filtered GRIT image-grounded captions; OCR examples cover recognition, comprehension, and reasoning.Source: §2.2.1 · pp. 4–5

3 EXPERIMENTS

Experiments evaluate VILA augmentation through self-recaptioning and specialist-generated data, then compare benchmark performance, caption quality, and costs. Self-augmentation gains level off after three rounds, while specialists extend improvements across benchmarks.

3 EXPERIMENTS

  • Experiments follow VILA’s three-stage alignment, pretraining, and instruction-tuning setup, replacing captions while keeping image sources fixed and evaluating common VLM benchmarks.The reported default setup uses 128 GPUs and a global batch size of 1024.Source: §3 · pp. 6–7

3.1 SELF-AUGMENTATION RESULTS

  • Self-augmentation progressively improves benchmark performance through three rounds, then saturates; later captions add richer details and more accurate visual descriptions.Caption length plateaus around rounds 3 and 4, while benchmark performance continues to improve through round 3.Source: §3.1 · p. 7, Table 3 · p. 8

3.2 SPECIALIST AUGMENTATION RESULTS

  • Specialist captions add object relations, localization, and clear pose; after the same SFT stage, spatial-specialist training improves 5 of 8 benchmarks, while combining three specialists yields significant improvements at full pretraining scale.Specialist-annotated data consistently improves results in both 10% and 100% pretraining settings.Source: §3.2 · pp. 7–8, Table 4 · p. 8

3.3 BENCHMARK COMPARISON TO PRIOR WORK

  • VILA2 exceeds VILA by noticeable margins across almost all benchmarks and reaches open-source SOTA on MMMU using publicly available datasets.The comparison covers 10 visual-language benchmarks.Source: §3.3 · p. 8, Table 6 · p. 9, Table 5 · p. 9

3.4 GAUGING ON SYNTHETIC DATA QUALITY AND HALLUCINATION

  • GPT-4V and Gemini prefer later-round captions; VSR results and blind human rankings indicate reduced hallucinations, with human preference saturating at Round-3.The human ranking compares recaptions of 200 images across 11 evaluators.Source: §3.4 · p. 9, Table 7 · p. 10, Table 8 · p. 10

3.5 IMPROVED DATA QUALITY MATTERS MORE THAN INCREASED COMPUTATION

  • Additional epochs on unchanged coarse image-text pairs yield no performance boosts despite extra costs, whereas improving data quality offers a more rewarding path.Source: §3.5 · p. 10, Table 9 · p. 10

3.6 EFFICIENCY AND EFFECTIVENESS OF VILA2

  • Recaptioning costs 0.12 USD per 1k images versus Amazon Turk’s 36 USD, a 300x reduction; the 51M-data training set is reusable across models.VILA2 processes around 38,340 images per hour; its recaptioned data is described as a one-time cost that can be shared.Source: §3.6 · p. 10

4 RELATED WORK & LIMITATIONS

The section situates VILA² among work that scales VLM pretraining through detector-generated labels and contrasts its self-augmentation approach with single-round GPT-4V distillation. It also states the method’s resource-bounded evaluation scope.

4 RELATED WORK & LIMITATIONS

  • Kosmos-2 and PaLI-X scaled pretraining with detector-pseudo-labeled bounding boxes, and examined how perception capabilities relate to VQA reasoning.The cited detectors are GLIP and OWL-v2, respectively.Source: §4 · p. 10
  • Unlike ShareGPT4V’s single GPT-4V recaptioning round, this work studies VLM self-augmentation without commercial APIs or distillation from larger models.The paper says its analysis covers prompts, iteration rounds, and saturation points.Source: §4 · p. 10
  • The comparison table covers multimodal methods across benchmarks and varies LLM and vision-tower parameters, image tokens, and pretraining-data size.Source: Table 10 · p. 11
  • The authors report efficient SOTA VLM performance enhancement without extra data or expensive closed-source APIs.Source: §4 · p. 11
  • Resource constraints limit the study to self-augmented curation and evaluations of 7B, 8B, and 40B models with 51M data.The authors identify larger models and more than 0.5B pretraining data as future work that could improve capabilities.Source: §4 · p. 11

5 CONCLUSIONS

The conclusion presents self-augmentation and specialist augmentation as two ways VLMs can improve their pretraining. It reports three feasible self-bootstrapping rounds, further gains from specialist knowledge distillation, and SOTA performance across benchmarks.

5 CONCLUSIONS

  • Three rounds of self-bootstrapping provide a feasible “free lunch” for VLMs, with further improvements from knowledge distillation using specialist VLMs.Source: §5 · p. 11
  • The VILA² models achieve SOTA performance across a comprehensive set of benchmarks.Source: §5 · p. 11
  • The authors identify deeper study of synergy between synthetic and real data as a future direction for training stronger foundation models.Source: §5 · p. 11

A.1 PROMPTS FOR SPECIALIST-AUGMENTATION

The specialist-augmentation prompts elicit image descriptions focused on spatial relations, grounded locations, or textual content. Each prompt specifies a different specialist task.

A.1 PROMPTS FOR SPECIALIST-AUGMENTATION

  • The spatial-relations prompt asks the specialist to elaborate on visual and narrative elements with a focus on spatial relations.Source: §A.1 · p. 18
  • The grounded-narration prompt asks for image details and locations specified as [xmin,ymin,xmax,ymax].Source: §A.1 · p. 18
  • The OCR prompt asks the specialist to recognize and describe image text in a brief description focused on textual content.Source: §A.1 · p. 18

A.2 SPECIALIST ACQUISITION

Specialist acquisition gathers task-specific knowledge from public datasets, filters noisy samples, and trains specialists from pretrained checkpoints. The authors also describe expertise sources and a difficulty with merging grounded narration.

A.2 SPECIALIST ACQUISITION

  • The pipeline gathers task-specific knowledge from public datasets, filters noisy samples with rule-based strategies, and trains specialist VLMs from pretrained checkpoints using different data blends and training strategies.Source: Figure 5 · p. 18
  • Specialist expertise can come from open-source data, human labels, or domain-specific models such as OCR and detection systems.Source: §A.2 · p. 18
  • For grounded narration, merging detector labels with language models introduced more hallucinations because duplicate labels complicate matching boxes to text.The authors report trying OWLv2, VLM-generated captions, and Llama3-70B-Instruct for this process.Source: §A.2 · p. 18

A.3 SFT DATA

The experiments use separate 1.8M-sample and 5.9M-sample SFT datasets. The 1.8M blend draws on visual-language instruction, captioning, and question-answering datasets.

A.3 SFT DATA

  • Experiments use a 1.8M-sample dataset for exploratory work and a 5.9M-sample dataset for state-of-the-art experiments.The SFT data-mixture table is identified as Table 11.Source: §A.3 · p. 18, Table 11 · p. 19
  • The 1.8M SFT Blend includes sources spanning captioning, visual question answering, OCR, visual reasoning, and instruction data.Listed sources include TextCaps, OCRVQA, CLEVR, VQAv2-train, and FLAN-1M.Source: §A.3 · p. 18
  • The 5.9M SFT Blend comprises datasets listed in the SFT data-mixture table.The passage identifies that table as Table 11 but does not provide its entries.Source: §A.3 · p. 18, Table 11 · p. 19

A.4 SPECIALIST DATA

Specialist data is combined with captioning and instruction-finetuning data to preserve narrative and instruction-following abilities while adding task-specific knowledge. Spatial, grounding, and OCR specialists use different domain-focused training mixtures.

A.4 SPECIALIST DATA

  • Specialty data is integrated with high-quality captioning and diverse instruction-finetuning data to retain narrative and instruction-following abilities while acquiring task-specific knowledge.Source: §A.4 · p. 19
  • The spatial specialist continues stage-2 training using ALLaVA captions and GPT-4V captions from ShareGPT4V.Source: §A.4 · p. 19
  • The grounding specialist trains in two stages, pairing short grounded narration with ALLaVA captions, then long narration with instruction and caption sources.The cleaned 4M grounded narration is split into Grounding-Short 3M and Grounding-Long 838K.Source: §A.4 · p. 19
  • The OCR specialist uses internet datasets for text recognition, understanding, and reasoning, including 35 OCR-related datasets from The Cauldron.Source: §A.4 · p. 19

A.5 TRAINING DETAIL

The training appendix describes staged training for 7B, 8B, and 13B models, a reduced-cost 40B setup, and shared hyperparameters. Additional caption comparisons report more detailed, less hallucinated recaptions across training rounds, with added spatial information from the spatial specialist.

A.5.1 7B & 8B & 13B MODELS

  • The 7B, 8B, and 13B training process has three stages: projector alignment, multimodal pretraining, and supervised finetuning.Pretraining replaces original COYO captions with augmented data; the final stage unfreezes all parameters for general-purpose VQA finetuning.Source: §A.5.1 · pp. 19–20

A.5.2 40B MODEL

  • The 40B model skips cost-intensive stage 2 and trains on 7.5M sampled COYO images paired equally with original, VILA3, and spatial-specialist VILA3 captions.Adding interleaved data such as MMC4 is left as future work.Source: §A.5.2 · p. 20

A.6 HYPERPARAMETERS

  • Training uses batch size 1024, cosine learning-rate decay, 0.03 warmup, no weight decay, and AdamW across 128 A100 GPUs.Table 13 is identified as the detailed training setup and hyperparameter table.Source: §A.6 · p. 20, Table 13 · p. 20

A.7 ADDITIONAL RE-CAPTION COMPARISONS

  • Across additional SAM caption comparisons, later VILA2 recaptioning rounds produce more detailed captions with fewer hallucinations, while the spatial specialist adds spatial information.The examples compare VILA2 rounds with InstructBLIP and LLaVA-NeXT-34B captions.Source: §A.7 · p. 20, Figure 6 · p. 21, Figure 7 · p. 22

Related papers

Loading 2407.17453v2…