PaperlayerView exact revision on arXiv

Source-linked AI summary

ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models

Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, Benyou Wang

arXiv:2402.11684v2cs.CLcs.AI

TL;DR

Lite LVLMs use fewer computational resources but exhibit a performance gap from normal-sized models. ALLaVA uses GPT-4V-synthesized training data, and its models outperform most 4B-scale VLMs across 17 benchmarks and perform on par with 7B-or-larger models.

  • Problem

    Whether high-quality training data can close the performance gap between resource-friendly lite LVLMs and normal-sized models remains an open question relevant to resource-limited use.

    Source: §1 · pp. 1–2

  • Method

    ALLaVA uses a Captioning-then-QA framework to prompt GPT-4V to generate fine-grained captions and visual question-answer pairs for a 1.3M-sample training dataset.

    Source: §1 · p. 2, §3.2 · p. 4, §6 · p. 9

  • Results

    Across 17 benchmarks, ALLaVA models outperform 4B-scale VLMs on most benchmarks and perform on par with 7B-or-larger VLMs.

    Source: §5.2.2 · p. 8

  • Takeaways & Limitations

    The results support the feasibility of using high-quality synthetic data to craft more efficient LVLMs.

    Source: §6 · p. 9

    The generated answers are not completely accurate, leaving answer quality in need of refinement.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large vision-language models (LVLMs) have shown premise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. However, they require considerable computational resources for training and deployment. This study aims to bridge the performance gap between traditional-scale LVLMs and resource-friendly lite versions by adopting high-quality training data. To this end, we propose a comprehensive pipeline for generating a synthetic dataset. The key idea is to leverage strong proprietary models to generate (i) fine-grained image annotations for vision-language alignment and (ii) complex reasoning visual question-answering pairs for visual instruction fine-tuning, yielding 1.3M samples in total. We train a series of lite VLMs on the synthetic dataset and experimental results demonstrate the effectiveness of the proposed scheme, where they achieve competitive performance on 17 benchmarks among 4B LVLMs, and even perform on par with 7B/13B-scale models on various benchmarks. This work highlights the feasibility of adopting high-quality data in crafting more efficient LVLMs. We name our dataset \textit{ALLaVA}, and open-source it to research community for developing better resource-efficient LVLMs for wider usage.

1 Introduction

Large vision-language models are costly to train and deploy, while lite models trade some performance for lower resource demands. ALLaVA investigates whether high-quality synthetic data can narrow this gap, introducing a GPT-4V data-generation framework, a 1.3M-sample dataset, and a series of 4B-scale models.

1 Introduction

  • Lite LVLMs are more accessible to users with scarce computation resources, but their performance falls short of normal-sized models.The introduction identifies the resource demands of training and deploying LVLMs as an obstacle to democratization.Source: §1 · pp. 1–2
  • The paper investigates whether scaling high-quality data can narrow the performance gap between normal-sized and lite LVLMs.It uses curated images from LAION and Vision-FLAN and proposes Captioning-then-QA to prompt GPT-4V to generate captions, instructions, and answers.Source: §1 · pp. 1–2
  • ALLaVA releases a 1.3M-sample synthetic dataset of fine-grained captions, complex instructions, detailed answers, and high-resolution images, alongside a series of trained 4B-scale LVLMs.The authors report that the models perform on par with larger models on multiple benchmarks.Source: §1 · p. 2

2 Rethinking Existing LVLMs

The section reassesses LVLMs from a data-centric perspective, focusing on alignment and visual instruction tuning: helping models recognize visual content and reason about it, then generalize across visual instructions.

2.1 On the Alignment

  • Caption-based alignment often relies on short, coarse captions that introduce noise, while existing COCO-based alternatives face limits in detail, scalability, or cross-modal association.BLIP’s CapFilt uses human-annotated COCO, whereas LLaVA’s GPT-4-generated conversations are bounded by COCO annotation detail and costly to scale; COCO image-text associations are also limited (Parekh et al., 2020).Source: §2.1 · p. 3

2.2 On Visual Instructions

  • Visual instruction datasets often ask relatively simple questions and emphasize fundamental abilities over advanced tasks such as complex reasoning.Vision-FLAN’s questions are described as simpler than WizardLM’s, while examples include object localization, book-cover text recognition, and one-sentence image captions.Source: §2.2 · p. 3
  • Vision-FLAN answers are often short, unformatted, and sometimes incomplete sentences, motivating their polishing or regeneration.The passage notes that learning these outputs directly could hinder model performance.Source: §2.2 · p. 3

3 Methodology of ALLaVA

ALLaVA uses GPT-4V to generate fine-grained captions and visual question-answering data, aiming to compensate for lite LVLMs’ reduced capacity. Manual inspection found its caption-then-answer strategy more accurate than direct answering, though less accurate than VFLAN ground truth.

3.2 The Philosophy of ALLaVA

  • ALLaVA targets lite LVLMs’ performance loss from fewer parameters by generating each image’s fine-grained caption and VQA pair in one GPT-4V session.The caption provides extra explicit context alongside image embeddings and may improve answer quality and reduce hallucination; lite LVLMs also offer lower training and deployment costs.Source: §3.2 · pp. 3–4, §3.1 · p. 3

3.3.1 Stage 0: Image Selection

  • Image selection draws on diverse web images from LAION and instruction-rich Vision-FLAN, with quality control and deduplication applied to images.Vision-FLAN combines 191 VQA tasks across 101 open-source datasets.Source: §3.3.1 · p. 4

3.3.2 Stage 1: Captioning

  • GPT-4V is prompted to describe multiple image aspects in detail, organizing the resulting fine-grained caption logically.The caption-generation prompt is illustrated in Figure 3.Source: §3.3.2 · p. 4

3.3.3 Stage 2: Questioning

  • For LAION images, GPT-4V generates five complex, diverse candidate questions and randomly selects one to answer; Vision-FLAN’s original instructions are retained.The intended questions require close image inspection and strong reasoning.Source: §3.3.3 · pp. 4–5

3.3.4 Stage 3: Answering

  • Answers are prompted to include relevant evidence, reasoning, and context, while LAION follows the full pipeline and Vision-FLAN retains its questions but receives regenerated answers.The authors argue that pure answers without context can make the input-output mapping unclear and may introduce hallucination.Source: §3.3.4 · p. 5

3.4 On the Ethics

  • Prompts require refusal of requests to disclose personal information or encourage discriminatory judgments.The paper identifies biased elements such as gender and race in descriptions of occupations as an ethical concern.Source: §3.4 · pp. 5–6

3.5 On the Effectiveness of Caption-then-QA Pipeline

  • Caption-then-Answer was 6% more accurate than Direct Answer but 4% below VFLAN gt in manual inspection of 100 VFLAN samples.The authors use this inspection to support the efficacy of the caption-then-QA curation pipeline.Source: §3.5 · p. 6, Table 1 · p. 6

3.6 The Resulted Dataset

  • The resulting datasets pair captions with VQA data for a 469K-image LAION subset and with answer-distilled instructions from Vision-FLAN.The sets are named ALLaVA-Caption-LAION-4V, ALLaVA-Instruct-LAION-4V, ALLaVA-Caption-VFLAN-4V, and ALLaVA-Instruct-VFLAN-4V.Source: §3.6 · p. 6

4 Exploring ALLaVA

ALLaVA is a large dataset built from diverse image sources and covering a wide range of topics. Its statistics and analyses characterize its scale, source diversity, and topic breadth.

Basic Statistics

  • 664K samples form ALLaVA, with average image resolution of 891 by 770 pixels; its Caption and Instruct subsets share the same images.The paper also states that ALLaVA surpasses other public datasets in multiple dimensions, including sample count and average resolution.Source: §4 · p. 7

Diverse Sources of Images in ALLaVA-LAION

  • 469K ALLaVA-LAION images come from 122K unique domains spanning news, templates, merchandise, and photography websites.The paper argues that these sources contribute high image quality and a variety of natural daily-life scenarios; Figure 7 shows the top-12 domain distribution.Source: §4 · p. 7, Figure 7 · p. 7

Topic Analysis

  • LDA analysis of 100K caption samples identified 25 topics, including natural scenery, fashion accessories, food ingredients, home decor, sports events, and vehicle parts.The researchers used Mallet-LDA, then asked GPT-4 to name clusters from their 10 most relevant words; the topic spread potentially improves model generalizability.Source: §4 · p. 7

5 Experiments

ALLaVA data improves existing lite VLMs across five benchmarks, and ALLaVA models outperform 4B-scale VLMs on most of 17 benchmarks while performing on par with 7B-or-larger models. Results also indicate gains in perception, cognition, and language ability.

5.1.1 Settings

  • The experiments add ALLaVA data to each training stage of MobileVLM-v2 and Mipha-3B and evaluate on MMMU, MMVP, TextVQA, LLaVA-Bench, and TouchStone.MobileVLM-v2 uses MobileLLaMA-2.7B-Chat with CLIP, while Mipha-3B uses Phi-2 with SigLIP; both use two-stage training.Source: §5.1.1 · pp. 7–8

5.1.2 Results and Analysis

  • Adding ALLaVA data to both training stages boosts MobileVLM-v2 and Mipha-3B on all five tested benchmarks, with especially large gains on MMVP, LLaVA-Bench, and TouchStone.The benchmarks cover perception as well as cognition and reasoning on complex queries.Source: §5.1.2 · p. 8

5.2.1 Settings

  • ALLaVA models use a LLaVA-v1.5-style architecture with a frozen CLIP-ViT-L/14@336 vision encoder and trainable projector and language-model backbone across two stages.Training mixes caption and textual data at pretraining, then VQA and textual data at finetuning; the evaluation covers one textual and 14 multimodal benchmarks.Source: §5.2.1 · p. 8

5.2.2 Results and Analysis

  • ALLaVA models outperform 4B-scale VLMs on most of 17 benchmarks and perform on par with 7B-or-larger models.Table 4 reports results on one textual and 16 multimodal benchmarks.Source: §5.2.2 · p. 8, Table 4 · p. 9
  • ALLaVA data compensates for the coarser-grained CLIP features relative to SigLIP, allowing ALLaVA models to outperform other 4B-scale models on several perception benchmarks.Source: §5.2.2 · p. 9
  • ALLaVA models outperform other 4B-scale models on 6/7 cognition benchmarks, especially in math, complex reasoning, and multimodal multidisciplinary knowledge.Source: §5.2.2 · p. 9
  • All ALLaVA models score higher than 4B-scale models on the textual benchmark; ALLaVA-Phi2 scores 49.4 versus 16.2 and 15.6 for same-backbone comparisons.ALLaVA-Phi3 performs best among the three lite ALLaVA models; the authors point to strong language backbones and textual data as important for avoiding catastrophic forgetting.Source: §5.2.2 · p. 9

6 Conclusion

ALLaVA introduces Caption-then-QA to generate high-quality captions, instructions, and answers, and releases a 1.3M-sample dataset for LVLM training. Ablations show that the dataset boosts lite VLM performance to parity with larger models on multiple benchmarks.

6 Conclusion

  • Ablation studies show that ALLaVA boosts lite VLMs to parity with larger VLMs on multiple benchmarks.Source: §6 · p. 9
  • Caption-then-QA is proposed as a simple, effective strategy for generating high-quality captions, instructions, and answers.Source: §6 · p. 9
  • ALLaVA is open-sourced as a 1.3M-sample dataset for LVLM training, described as the largest dataset of its kind so far.Source: §6 · p. 9

7 Limitation … A.3 Prompt for Distilling Vision-FLAN

The paper notes that synthetic answers remain imperfect and that minority-culture multilingual content is limited. Its appendices describe LAION filtering and prompts that elicit detailed image descriptions and image-grounded answers, with safeguards for sensitive content.

7 Limitation

  • Caption-then-QA improves accuracy over directly generating responses, but answers remain imperfect and minority-culture multilingual content is not fully covered.The authors identify both as limitations and areas for future refinement.Source: §7 · p. 9

A.1 Deduplication on LAION Samples

  • LAION-400M images were filtered by resolution, potentially pornographic or violent URLs, and caption-embedding similarity below 440 for deduplication.The pipeline used all-mpnet-base-v24 embeddings and excluded images with both dimensions exceeding 512 pixels.Source: §A.1 · p. 15

A.2 Prompt for Distilling LAION

  • The LAION prompt requests a detailed image description, then five diverse questions requiring close inspection and reasoning, randomly selecting one to answer solely from the image.The question should concern valuable information in the picture.Source: §A.2 · p. 15
  • The LAION prompt rules out questions involving violence, advertising, privacy invasion, or discomfort and specifies tagged output fields for description, candidate questions, selected question, and answer.The response must not mention the prompt itself.Source: §A.2 · p. 15

A.3 Prompt for Distilling Vision-FLAN

  • The Vision-FLAN prompt asks first for an image description covering subjects, background, colors, notable features, context, and any emphasized elements.The output format brackets the description with start and end markers.Source: §A.3 · pp. 15–16
  • The Vision-FLAN answer should address the question with relevant information and demonstrate the solving process, using the detailed-answer output field.The prompt supplies the question in a designated field.Source: §A.3 · p. 16
  • The Vision-FLAN prompt calls for professional, helpful, objective, unbiased responses and unbiased descriptions of traits such as gender and race in traditionally biased scenarios.It gives occupation references as an example of such scenarios.Source: §A.3 · p. 16
  • The Vision-FLAN prompt requires refusing questions that seek personal information leaks or discriminatory judgments against underrepresented groups.It directs the model to identify the inappropriate intent.Source: §A.3 · p. 16

A.4 Data Example · A.5 Manual Inspection on Subsets of VFLAN

The appendix illustrates the generated datasets with examples and reports a manual inspection of VFLAN subsets, which found that captioning before answering reduced hallucinations. The inspection illustrates this with a blur image that was misinterpreted without a caption but correctly identified as defocus blur with one.

A.4 Data Example

  • Table 5 presents two examples from each dataset, pairing an image with its caption, question, and answer.The entries in bold were generated by GPT-4V using the data-generation protocol.Source: §A.4 · p. 16
  • Boldface in the examples identifies entries generated by GPT-4V using the authors’ data-generation protocol.These examples appear in Table 5.Source: §A.4 · p. 16

A.5 Manual Inspection on Subsets of VFLAN

  • Captioning the image before answering reduced hallucinations in the manual inspection of generated data.The inspection results are shown in Table 1.Source: §A.5 · p. 16
  • Without prior captioning, the model interpreted image blur as directional and attributed it to motion, whereas caption-first answering identified defocus blur.The passage presents this contrast in Figure 9.Source: §A.5 · p. 16
  • The manual inspection uniformly sampled 100 data pieces across the original VFLAN categories to assess generated-data quality.The passage says the inspection results are shown in Table 1.Source: §A.5 · p. 16

B Details of Topic Analysis … C Training details

The supplied passages describe GPT-4-based topic naming from clustered keywords and provide prompts and visuals documenting topic generation and examples. They do not provide training details.

B.1 Prompt for Topic Name Generation

  • GPT-4 generates a topic name from ten words per cluster using the prompt shown in Figure 10.Source: §B.1 · p. 16, Figure 10
  • Table 5 presents data examples, with bold entries generated by GPT-4V using the data-generation protocol.Source: Table 5 · p. 17

B.2 Full List of Topics

  • Figure 9 compares answers without and with captioning, marking errors in red and correct answer portions in green for visualization.Source: Figure 9 · p. 18
  • The topic-naming prompt asks GPT-4 to summarize LDA-derived words in one or two words and output only the summary.Source: §B.2 · p. 18

B.1 Prompt for Topic Name Generation

  • Figure 10 shows the prompt used to generate topic names.Source: Figure 10

B.2 Full List of Topics

C.1 Training data … D Evaluation

The appendix identifies the datasets used for ALLaVA training and points to tables documenting model hyperparameters and training time; experiments ran on a single 8*A800 GPU node.

C.1 Training data

  • ALLaVA training uses OpenChat for text, ShareGPT4V for captions, and llava_instruct_657K for VQA.Source: Table 7 · p. 19

C.2 Hyperparameters

  • ALLaVA model training hyperparameters are detailed in Table 8.Source: §C.2 · p. 19

C.3 Computation Resources

  • All experiments ran on a single 8*A800 GPU node, and model training times are shown in Table 9.Source: §C.3 · p. 19

D.1 Details on Benchmarks · D.2 Evaluation Prompt for Vicuna-80

The appendix specifies the benchmarks and their scoring protocols, then gives the pairwise answer-selection prompt used for Vicuna-80 evaluation.

D.1 Details on Benchmarks

  • GQA has 12,578 reasoning and compositional questions; HallusionBench has 254 samples; MME spans 14 subtasks and 2,374 questions; MMVP tests “CLIP-blind” pairs, all using accuracy.Source: §D.1 · p. 19
  • TouchStone contains 908 open-ended questions across 5 abilities and 27 subtasks; text-based GPT-4 judges answers, with averaged scores as the metric.Source: §D.1 · pp. 19–20
  • TextVQA has 5,000 questions and uses accuracy; MathVista has 6,141 samples for mathematical reasoning; MM-Vet has 218 multi-capability questions and uses accuracy.Source: §D.1 · p. 20
  • The MMMU validation set has 900 expert-level multiple-choice questions, while ScienceQA has 4,201 questions across subjects and categories; both use accuracy.Source: §D.1 · p. 20
  • LLaVA-Bench (In-the-Wild) uses 60 open-ended questions and GPT-4 pairwise judging, scored by candidate-to-anchor score ratio; MLLM-Bench uses 420 questions and anchor win rate.Source: §D.1 · p. 20
  • MMBench’s dev set has 4,329 multiple-choice questions across 20 ability dimensions; SEED-Bench-v1’s image set has 14,233 across 9 dimensions, both scored by accuracy.Source: §D.1 · p. 20

D.2 Evaluation Prompt for Vicuna-80

  • The Vicuna-80 evaluation prompt asks a judge to compare two answers to a question and return either “Answer1” or “Answer2”.Source: §D.2 · p. 21

Related papers

Loading 2402.11684v2…