PaperlayerView exact revision on arXiv

Source-linked AI summary

ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models

Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, Benyou Wang

arXiv:2402.11684v2cs.CLcs.AI

TL;DR

Lite vision-language models use fewer resources but trail normal-sized models in performance. ALLaVA addresses this gap with a GPT-4V-based Caption-then-QA pipeline that produces 1.3M synthetic samples, and its models outperform 4B-scale systems on most of 17 benchmarks while matching larger models on multiple benchmarks.

  • Problem

    Lite LVLMs reduce computational demands but retain a performance gap relative to normal-sized models, motivating whether high-quality data can close it.

  • Method

    ALLaVA uses a Caption-then-QA framework that prompts GPT-4V to generate fine-grained captions, questions, and answers for curated images.

  • Results

    ALLaVA models outperform 4B-scale LVLMs on most of 17 benchmarks and perform on par with 7B or larger models on multiple benchmarks.

  • Takeaways & Limitations

    The results support the feasibility of using high-quality synthetic data to improve the performance of resource-efficient LVLMs.

  • Takeaways & Limitations

    The generated answers are not completely accurate, and the current dataset does not fully cover multilingual content related to minority cultures.

Abstract

from arXiv · show

Large vision-language models (LVLMs) have shown premise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. However, they require considerable computational resources for training and deployment. This study aims to bridge the performance gap between traditional-scale LVLMs and resource-friendly lite versions by adopting high-quality training data. To this end, we propose a comprehensive pipeline for generating a synthetic dataset. The key idea is to leverage strong proprietary models to generate (i) fine-grained image annotations for vision-language alignment and (ii) complex reasoning visual question-answering pairs for visual instruction fine-tuning, yielding 1.3M samples in total. We train a series of lite VLMs on the synthetic dataset and experimental results demonstrate the effectiveness of the proposed scheme, where they achieve competitive performance on 17 benchmarks among 4B LVLMs, and even perform on par with 7B/13B-scale models on various benchmarks. This work highlights the feasibility of adopting high-quality data in crafting more efficient LVLMs. We name our dataset \textit{ALLaVA}, and open-source it to research community for developing better resource-efficient LVLMs for wider usage.

1 Introduction

LVLMs offer broad vision-language capabilities but demand substantial training and deployment resources, motivating lite models that sacrifice performance. ALLaVA addresses this gap by scaling high-quality synthetic data and releasing 1.3M samples plus 4B-scale models that perform competitively with larger LVLMs.

  • Motivation: Lite LVLMs reduce computation requirements and improve portability, but generally incur performance losses relative to normal-sized models.These losses limit their ability to match the capabilities of larger LVLMs.
  • Motivation: The work investigates whether scaling high-quality data can close the performance gap between normal-sized and lite LVLMs.It curates high-quality images from LAION and Vision-FLAN and proposes a data-generation framework.
  • Contributions: ALLaVA open-sources 1.3M diverse synthetic samples with GPT-4V-generated fine-grained captions, complex instructions, detailed answers, and high-resolution images.The images are curated from different sources for LVLM training.
  • Contributions: ALLaVA releases a series of 4B-scale LVLMs trained on the synthetic dataset that perform on par with larger models across multiple benchmarks.The reported results support the effectiveness of high-quality synthetic data for resource-efficient LVLM development.

2 Rethinking Existing LVLMs

The paper rethinks existing LVLMs from a data-centric perspective, focusing on alignment and visual instruction tuning. It identifies coarse-grained image-text alignment, relatively simple questions, and short, uninformative answers as key limitations.

  • Data-centric Perspective: The analysis reevaluates LLaVA Liu et al. (2023a) by scrutinizing both its alignment and visual instruction tuning stages under a data-quality principle.Alignment supports visual understanding and reasoning, whereas instruction tuning aims to improve generalization across diverse visual instructions.
  • Image-text Alignment: Existing caption datasets provide short, coarse-grained descriptions that introduce noisy signals and hinder vision-language alignment.The alignment stage is intended to help language models recognize visual objects and improve visual reasoning.
  • Visual Instruction Tuning: Visual instruction tuning datasets often emphasize fundamental abilities, while their questions remain relatively simple compared with WizardLM Xu et al. (2023a).Vision-FLAN Xu et al. (2023b), for example, contains 191 VQA tasks across 101 datasets, but its questions are described as relatively simple.
  • Visual Instruction Tuning: Vision-FLAN answers are frequently short, uninformative, or incomplete, so directly learning them can hinder model performance and motivates answer polishing or regeneration.The answers are manually annotated but often consist of words or phrases without format prompts, and some are incomplete sentences.

3 Methodology of ALLaVA

ALLaVA synthesizes fine-grained captions and complex visual question-answering data by prompting GPT-4V through image selection, captioning, questioning, and answering stages. The Caption-then-Answer strategy improves answer accuracy over direct answering, while combining LAION and Vision-FLAN sources supports diverse synthetic datasets.

  • Data synthesis: ALLaVA prompts GPT-4V to generate a fine-grained caption and a VQA pair for each image within one session, using captioning, questioning, and answering stages.The caption supplies additional explicit image context alongside implicit image embeddings, supporting answer quality and reducing hallucination.
  • Image selection: Images come from LAION and Vision-FLAN, with quality control and deduplication applied to support diverse, real-world-aligned image sources.Questions are generated only for LAION, while Vision-FLAN’s original instructions are retained because they already provide diverse instructions.
  • Captioning, questioning, and answering: GPT-4V is prompted to produce detailed, logically organized captions, complex and diverse questions, and answers containing evidence, chain-of-thought, and relevant context.The question-generation prompt uses lightweight complexity instructions rather than WizardLM’s original instruction evolution, and the answer prompt avoids pure answers without context.
  • Data generation evaluation: 6% higher accuracy than Direct Answer, though 4% below VFLAN gt, validates the Caption-then-Answer curation pipeline in manual inspection.The inspection evaluated 100 VFLAN samples by directly generating answers with or without captioning first.
  • Synthetic datasets: The pipeline creates ALLaVA-Caption and ALLaVA-Instruct datasets from 469K LAION images and Vision-FLAN instructions, regenerating VFLAN answers while retaining its questions.LAION questions compensate for VFLAN’s simpler questions, whereas original VFLAN questions supplement overall question diversity.

4 Exploring ALLaVA

Section 4 characterizes ALLaVA’s scale, image-source diversity, and topic coverage. The dataset contains 664K samples spanning diverse images and is built from broad web-domain sources and topic clusters.

  • Basic Statistics: ALLaVA contains 664K samples with average image resolutions of 891 by 770 pixels, divided into ALLaVA-Caption and ALLaVA-Instruct subsets sharing the same images.The dataset is reported to surpass other public datasets in multiple dimensions, including sample count and average resolution.
  • Diverse Sources of Images in ALLaVA-LAION: ALLaVA-LAION’s 469K images come from 122K unique domains spanning news, high-resolution templates, merchandise, and high-resolution photography websites.Figure 7 illustrates the distribution of the top-12 domains.
  • Topic Analysis: Topic diversity is analyzed across ALLaVA-Caption-LAION-4V and ALLaVA-Caption-VFLAN-4V using 100K samples, 25 Mallet-LDA topics, 1K training steps, and GPT-4-generated cluster summaries.For each cluster, the ten most relevant words are supplied to GPT-4 using prompts described in Appendix B.1.

5 Experiments

Experiments show that ALLaVA improves existing lite VLMs and enables models that outperform most 4B-scale VLMs across 17 benchmarks, with performance comparable to larger models. Results further highlight gains in perception, cognition, and language ability from ALLaVA data and strong language backbones.

  • Ablation: Adding ALLaVA data to both training stages boosts MobileVLM-v2 and Mipha-3B on all five evaluated benchmarks, especially MMVP, LLaVA-Bench, and TouchStone.These benchmarks probe perception, cognition, and reasoning abilities.
  • Overall Results: ALLaVA models outperform 4B-scale VLMs on most of 17 benchmarks and perform on par with 7B-or-larger VLMs.The comparison covers 1 textual and 16 multimodal benchmarks.
  • Multimodal Perception: ALLaVA models outperform other 4B-scale models on several perception benchmarks despite CLIP’s coarser features than SigLIP, which ALLaVA’s fine-grained data compensates for.The authors attribute perception performance to the interaction between vision-encoder representations and fine-grained training annotations.
  • Multimodal Cognition: ALLaVA models outperform other 4B-scale models on 6/7 cognition benchmarks, particularly for mathematics, complex reasoning, and multimodal multidisciplinary knowledge.The models use a LLaVA-v1.5-style architecture with CLIP-ViT-L/14@336 and lightweight language backbones.
  • Language Ability: On the textual benchmark, ALLaVA-Phi2 scores 49.4 versus 16.2 for Mipha-3B and 15.6 for TinyLLaVA, while ALLaVA-Phi3 performs best among the three lite models.The results indicate the importance of a strong language backbone and textual data after visual instruction tuning.

6 Conclusion

The paper introduces Caption-then-QA for generating high-quality captions, instructions, and answers, and releases ALLaVA, a 1.3M-sample LVLM training dataset. Ablations show that ALLaVA boosts lite VLMs to parity with larger VLMs across multiple benchmarks.

  • 6 Conclusion: Caption-then-QA is proposed as a simple yet effective strategy for generating high-quality captions, instructions, and answers.
  • 6 Conclusion: ALLaVA is open-sourced as the largest reported LVLM training dataset, containing 1.3M samples.
  • 6 Conclusion: Ablation studies show that ALLaVA improves lite VLM performance to match larger VLMs on multiple benchmarks.

7 Limitation … A.3 Prompt for Distilling Vision-FLAN

The paper identifies remaining data-quality and coverage limitations, then documents filtering and prompt designs for distilling LAION and Vision-FLAN data. The prompts require detailed image descriptions, reasoning-based answers, structured outputs, and ethical safeguards.

  • 7 Limitation: The Caption-then-QA strategy improves accuracy over directly generating responses, but its answers remain imperfect and multilingual coverage of minority cultures is incomplete.The authors identify both answer accuracy and culturally inclusive multilingual data as areas for future refinement.
  • A.1 Deduplication on LAION Samples: LAION images are filtered by resolution, potentially pornographic or violent URLs, and semantic-caption similarity to remove duplicates.The process uses Laion-400M images, excludes images with both dimensions above 512 pixels, and applies all-mpnet-base-v24 caption embeddings for semantic retrieval.
  • A.2 Prompt for Distilling LAION: The LAION distillation prompt asks for a detailed image description, five diverse candidate questions, one randomly selected question, and an image-grounded answer.It also directs the model to target valuable image information and avoid violence, advertisements, privacy invasion, or discomfort.
  • A.2 Prompt for Distilling LAION: The LAION prompt enforces a structured output containing description, candidate questions, selected question, and answer fields.The response must not mention the prompt and must follow explicit start-and-end delimiters for each field.
  • A.3 Prompt for Distilling Vision-FLAN: The Vision-FLAN prompt first requests an image description covering subjects, background, colors, notable features, and context, then requires a relevant answer with its solving process.Descriptions may emphasize specified image elements, while both description and answer should be professional, insightful, helpful, objective, and unbiased.
  • A.3 Prompt for Distilling Vision-FLAN: The Vision-FLAN prompt requires unbiased specification of traits such as gender and race and refusal of requests involving personal-information leaks or discrimination.It frames these requirements around scenarios where bias has traditionally been an issue and questions that violate ethical rules.
  • A.3 Prompt for Distilling Vision-FLAN: Vision-FLAN outputs use explicit delimiters for a description and a detailed answer around the supplied question.The prompt presents the question in a dedicated placeholder and specifies the required output format.

A.4 Data Example · A.5 Manual Inspection on Subsets of VFLAN

The paper illustrates generated VFLAN examples and manually inspects sampled data to assess generation quality. The inspection indicates that captioning images before answering reduced hallucinations.

  • A.4 Data Example: Table 5 presents two examples from each dataset, pairing an image with its caption, question, and answer.The examples expose the structure of the generated data across datasets.
  • A.4 Data Example: The displayed examples identify GPT-4V-generated entries in bold.This formatting distinguishes generated content within the examples.
  • A.4 Data Example: The examples demonstrate a data-generation protocol spanning image descriptions, questions, and answers.Each showcased entry contains all three components alongside the image.
  • A.5 Manual Inspection on Subsets of VFLAN: The authors uniformly sampled 100 data pieces from the original VFLAN categories for detailed manual inspection.The inspection was designed to verify the quality of the generated data.
  • A.5 Manual Inspection on Subsets of VFLAN: Table 1 reports the results of the manual inspection across the sampled VFLAN data.The inspection follows the original category distribution through uniform sampling.
  • A.5 Manual Inspection on Subsets of VFLAN: Captioning the image before answering reduced the model’s hallucinations by providing a detailed image description beforehand.Figure 9 is cited in connection with answers produced without prior captioning, but the supplied passage does not report its quantitative values.

B Details of Topic Analysis … C Training details

The appendix details GPT-4-based topic-name generation from cluster keywords and documents the resulting topics, data examples, and captioning-related answer comparisons.

  • B.1 Prompt for Topic Name Generation: GPT-4 generates a topic name from ten keywords provided for each cluster using the prompt shown in Figure 10.The topic-analysis procedure uses cluster-level keyword lists as input to GPT-4.
  • B.1 Prompt for Topic Name Generation: Table 5 illustrates generated data examples, with bold entries indicating content produced by GPT-4V through the data-generation protocol.The examples show outputs associated with the paper’s synthetic-data process.
  • B.2 Full List of Topics: One topic example asks the model to identify the blur type in a blurred image, offering zoom, glass, motion, and defocus blur options.The example frames blur identification as a multiple-choice visual question.
  • B.2 Full List of Topics: Figure 9 compares answers produced without captioning against answers produced with captions, marking errors in red and correct portions in green.The color markings are intended only to improve visualization of the comparison.
  • B.1 Prompt for Topic Name Generation: Figure 10 presents the prompt used to generate topic names from cluster keywords.This visual corresponds to the GPT-4 topic-naming procedure described in the appendix.
  • B.2 Full List of Topics: Table 6 provides the full list of topics generated by LDA and GPT-4.The table documents topics produced by both topic-generation methods.

C.1 Training data … D Evaluation

The supplied appendix passages identify ALLaVA’s training datasets, hyperparameter documentation, and computation setup, but do not provide substantive evaluation findings. Training uses OpenChat for text, ShareGPT4V for captions, and llava_instruct_657K for VQA.

  • C.1 Training data: Table 7 summarizes the datasets used to train ALLaVA models.
  • D Evaluation: The supplied passages do not report substantive findings from D Evaluation.
  • C.1 Training data: ALLaVA training combines OpenChat for text, ShareGPT4V for image captions, and llava_instruct_657K for visual question answering.These dataset assignments are summarized in Table 7.
  • C.2 Hyperparameters: The training hyperparameters for ALLaVA models are documented in Table 8.
  • C.3 Computation Resources: All experiments run on a single 8*A800 GPU node.Training time for each model is reported in Table 9.
  • C.3 Computation Resources: Table 9 reports the training time of each ALLaVA model.

D.1 Details on Benchmarks · D.2 Evaluation Prompt for Vicuna-80

The evaluation uses a broad suite of vision-language benchmarks spanning reasoning, hallucination, mathematical ability, expert knowledge, and multimodal capabilities, with task-specific metrics. Vicuna-80 is evaluated by prompting a judge to choose the better of two candidate answers.

  • D.1 Details on Benchmarks: The study’s evaluation benchmarks are explicitly detailed as the basis for assessing the models across these varied tasks and capabilities.The listed benchmark abbreviations include Vicuna-80, GQA, HallusionBench, MME, MMVP, TouchStone, TextVQA, MathVista, MMMU, ScienceQA, LLaVA-Bench, MLLM-Bench, MMBench, and SEED-Bench-v1.
  • D.1 Details on Benchmarks: The benchmark suite covers 12,578-question GQA, 254-sample HallusionBench, 2,374-question MME, and CLIP-blind-pair MMVP, primarily using accuracy.GQA tests real-world reasoning and compositional question answering; HallusionBench evaluates hallucination; MME spans 14 subtasks; and MMVP tests “CLIP-blind” pairs.
  • D.1 Details on Benchmarks: TouchStone contains 908 open-ended questions across 5 abilities and 27 subtasks, and scores answers against pre-generated text-based GPT-4 answers using averaged scores.The benchmark uses text-based GPT-4 as the judge.
  • D.1 Details on Benchmarks: The suite also includes TextVQA with 5,000 questions, MathVista with 6,141 samples, MM-Vet with 218 multi-capability questions, MMMU validation with 900 expert-level questions, and ScienceQA with 4,201 questions.Accuracy is used for TextVQA, MM-Vet, MMMU, and ScienceQA; MathVista evaluates mathematical reasoning ability.
  • D.1 Details on Benchmarks: Additional evaluations comprise LLaVA-Bench’s 60 open-ended questions, MLLM-Bench’s 420 complex visual reasoning questions, MMBench’s 4,329-question dev set, and SEED-Bench-v1’s 14,233-question image set.These use GPT-4 pairwise score ratios, GPT-4V-judged win rates over LLaVA-v1.5-13B anchors, circular-evaluation accuracy, and accuracy, respectively.
  • D.2 Evaluation Prompt for Vicuna-80: For Vicuna-80, the evaluator receives a question and two candidate answers and must judge which answer is of better quality.The prompt inserts the question and both answers into explicit placeholders.
  • D.2 Evaluation Prompt for Vicuna-80: The Vicuna-80 judge must output only either “Answer1” or “Answer2.”This constrains evaluation to a binary preference between the presented responses.
Loading 2402.11684v2…