PaperlayerView exact revision on arXiv

Source-linked AI summary

VIGC: Visual Instruction Generation and Correction

Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, Conghui He

arXiv:2308.12714v3cs.CVcs.AI

TL;DR

High-quality vision-language instruction data is scarce, and language-only generation requires pre-annotations while accessible multimodal models may produce inadequate or false answers. VIGC generates diverse visual question-answer data and iteratively corrects it. The authors report improved benchmark performance and data quality validated through evaluations.

  • Problem

    High-quality multimodal instruction data is difficult to acquire, while language-only GPT-4 generation depends on costly manual pre-annotation and accessible MLLMs can produce inadequate or false responses.

    Source: Introduction · p. 1

  • Method

    VIGC uses VIG to generate visual question-answer pairs and VIC's iterative update strategy to mitigate hallucinations and produce high-quality instruction data.

    Source: Introduction · p. 2

  • Results

    VIGC-trained models improved across MMBench, OKVQA, and A-OKVQA, and LLaVA-7B surpassed LLaVA-13B.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    VIGC generated diverse instruction-following data on COCO and Objects365, and the authors report that correction significantly reduced hallucinations.

    Source: Conclusion · p. 7

    Some hallucinations persist after Visual Instruction Correction.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

The integration of visual encoders and large language models (LLMs) has driven recent progress in multimodal large language models (MLLMs). However, the scarcity of high-quality instruction-tuning data for vision-language tasks remains a challenge. The current leading paradigm, such as LLaVA, relies on language-only GPT-4 to generate data, which requires pre-annotated image captions and detection bounding boxes, suffering from understanding image details. A practical solution to this problem would be to utilize the available multimodal large language models (MLLMs) to generate instruction data for vision-language tasks. However, it's worth noting that the currently accessible MLLMs are not as powerful as their LLM counterparts, as they tend to produce inadequate responses and generate false information. As a solution for addressing the current issue, this paper proposes the Visual Instruction Generation and Correction (VIGC) framework that enables multimodal large language models to generate instruction-tuning data and progressively enhance its quality on-the-fly. Specifically, Visual Instruction Generation (VIG) guides the vision-language model to generate diverse instruction-tuning data. To ensure generation quality, Visual Instruction Correction (VIC) adopts an iterative update mechanism to correct any inaccuracies in data produced by VIG, effectively reducing the risk of hallucination. Leveraging the diverse, high-quality data generated by VIGC, we finetune mainstream models and validate data quality based on various evaluations. Experimental results demonstrate that VIGC not only compensates for the shortcomings of language-only data generation methods, but also effectively enhances the benchmark performance. The models, datasets, and code are available at https://opendatalab.github.io/VIGC.

Introduction

The paper addresses the difficulty of obtaining high-quality multimodal instruction data when language-only generation depends on manual annotations and accessible multimodal models can produce inadequate or false answers. It proposes VIGC to generate diverse visual question-answer data, correct inaccuracies iteratively, and evaluate the resulting data through downstream training.

Introduction

  • High-quality multimodal instruction data is harder to acquire than pretraining data, while language-only GPT-4 generation requires manual pre-annotation and is limited to annotated information.Questions outside that information cannot be answered, and answerable questions lose image detail.Source: Introduction · pp. 1–2
  • Existing VLM-based data generation faces a quality-control gap, as accessible MLLMs can produce inadequate responses and hallucinations.The paper notes that prior approaches did not address generated-data quality or validate it experimentally.Source: Introduction · p. 2
  • VIGC uses VIG to generate diverse visual question-answer pairs and VIC to reduce hallucinations through iterative updates.The framework is intended to autonomously generate high-quality image-text instruction fine-tuning data.Source: Introduction · p. 2
  • The authors release VIGC-LLaVA-COCO with 36,781 examples and VIGC-LLaVA-Objects365 with approximately 1.8 million examples.They describe these as datasets generated using VIGC for research on large multimodal models.Source: Introduction · p. 2
  • VIGC-generated data improved LLaVA-7B enough to surpass LLaVA-13B, and models trained with it improved across MMBench, OKVQA, and A-OKVQA.The experiments assess the generated data through model training and multimodal evaluations.Source: Introduction · p. 2

Related Work

Prior instruction-tuning approaches range from language-only instruction datasets to multimodal datasets built from annotated or existing image-text data. VIGC is framed as a self-guided alternative that generates diverse image-related questions and answers for novel images.

Instruction-following LLMs

  • Instruction-following language-model datasets often derive from pre-existing NLP tasks, which limits their generalizability to multimodal data.SELF-INSTRUCT uses generated instruction data to enhance LLM performance, but these methods do not directly generalize to multimodal data.Source: Instruction-following LLMs · p. 2

Multi-modal Instruction Tunning

  • Multimodal instruction datasets have relied on human annotations or existing data, limiting scalability, diversity, or volume; VIGC instead targets model-driven data generation for novel images.MiniGPT-4 curates 3,500 image-text pairs, LLaVA requires manual annotations, and InstructBLIP converts 26 datasets into instruction data.Source: Multi-modal Instruction Tunning · pp. 2–3

Visual Question Generation

  • Unlike conventional visual question generation, which focuses on relevant questions from images, VIGC develops diverse questions and appropriate answers across requirement categories.The paper says VIGC leverages large language model knowledge and is not limited by the training sample sizes typical of traditional VQG.Source: Visual Question Generation · p. 3

Methods

VIGC uses existing visual instruction data and templates to train a generation-and-correction framework. VIG produces instruction-conditioned image question-answer pairs, while iterative VIC updates address hallucinations in generated answers.

Methods

  • VIGC uses existing fine-tuning data to guide generation of higher-quality, more diverse visual instruction data.The framework is intended to mitigate challenges in generating multimodal instruction data with existing models.Source: Methods · p. 3

Initial Instruction Construction

  • The initial dataset combines visual instruction-tuning examples with templates for dialogue, detailed description, and complex reasoning.Each example pairs an image and instruction with a related question and answer for training type-conditioned generation.Source: Initial Instruction Construction · pp. 3–4

Visual Instruction Generation

  • VIG uses visual features, instruction-aware Q-Former processing, and a language model to generate image-linked question-answer pairs.VIG and VIC share network parameters, differing primarily in the data used for training.Source: Visual Instruction Generation · p. 4

Visual Instruction Correction

  • VIC counters hallucinations by conditioning answers on the question and iteratively updating Q-Former visual features with recent answer text.The authors report this is beneficial for detailed descriptions but relatively limited for dialogue and inference tasks.Source: Visual Instruction Correction · pp. 4–5

Experiments

The experiments evaluate VIGC-generated instruction data across COCO and Objects365, LLaVA-style evaluations, and knowledge-intensive VQA tasks. Adding generated data improves results in the reported settings, including after VIC refinement.

Datasets

  • The generation setup uses LLaVA-150K and OKVQA/A-OKVQA instruction data for training, then generates data from COCO and Objects365 while excluding test-set images.COCO and Objects365 test generation within the same and different image domains.Source: Datasets · p. 5

Implementation Details

  • Training starts from MiniGPT-4+ parameters and fine-tunes its Q-Former and projection layer while keeping the language and visual model parameters fixed.The best-performing checkpoint across 10 epochs is selected for data generation.Source: Implementation Details · p. 5

LLaVA Data and Evaluation

  • Conversation and complex-reasoning data yield appreciable gains, while detailed-description data alone gives only marginal improvement amid severe illusions.Applying VIC before training on all three data types produces additional improvements.Source: LLaVA Data and Evaluation · p. 6, Table 1 · p. 6
  • Generated data from COCO or Objects365 improves performance across domains, and gains are also reported for LLaVA-13B.The authors describe these results as a cross-domain improvement and cite Table 3 for the comparisons.Source: LLaVA Data and Evaluation · p. 7, Table 3 · p. 7
  • Fine-tuning VIGC again on 36K COCO-generated examples improves performance on both MMBench and LLaVA.The authors describe this as self-iterative training.Source: LLaVA Data and Evaluation · p. 7, Table 2 · p. 7

OK-VQA Dataset and Evaluation

  • 0.7% on OKVQA and 1.6% on A-OKVQA: InstructBLIP fine-tuned with generated data outperforms fine-tuning with original data alone.The authors report state-of-the-art results for models of this scale on both datasets.Source: OK-VQA Dataset and Evaluation · p. 7, Table 4 · p. 7

Conclusion

VIGC autonomously generates vision-language instruction data on COCO and Objects365, and evaluations validate its quality. VIC significantly reduces hallucinations, though some persist.

Conclusion

  • VIGC provides a self-instruct method for autonomously generating high-quality vision-language instruction data.The framework generates diverse multimodal instruction-following data on COCO and Objects365, with quality assessed through varied evaluations.Source: Conclusion · p. 7
  • VIC significantly reduces hallucinations, but some instances remain.The authors identify eliminating multimodal hallucinations and exploring a closed-loop generation-and-training system as future work.Source: Conclusion · p. 7

A. Instruction Templates

The instruction templates cover conversation, detailed description, complex reasoning, and outside-knowledge visual question answering.

A. Instruction Templates

  • The templates are organized by four task types: conversation, detailed description, complex reasoning, and outside-knowledge visual question answering.The paper lists the corresponding templates in Tables 5–8.Source: A. Instruction Templates · p. 10, Table 5 · p. 10, Table 6 · p. 10
  • The hallucination figure distinguishes model limitations, training-data bias, and information decay in sequence generation.Table 9 defines its statistics as hallucination count, counts in the first and second sentence halves, and hallucinated words.Source: Figure 5 · p. 11, Table 9 · p. 11

B. Hallucination analysis for the VIGC model.

VIGC attributes hallucinations to visual-model limitations, training-data bias, and information decay during sequence generation. Its iterative correction reduces measured hallucination in generated data.

B. Hallucination analysis for the VIGC model.

C. Data Statistical Analysis

The analysis reports statistics for VIGC-generated dialogue, detailed-description, and complex-reasoning data from COCO and Objects365 images, including question diversity and question-word distributions.

C. Data Statistical Analysis

  • VIGC-generated data statistics cover dialogue, detailed description, and complex reasoning using COCO and Objects365 images.Table 10 presents sample data, average question and answer lengths, and question diversity.Source: C. Data Statistical Analysis · p. 12, Table 10 · p. 12
  • Dialogue questions often begin with “what is” or “what color,” while complex-reasoning questions often begin with “what might be” or “what can be.”Figure 6 visualizes the frequency of questions beginning with different words.Source: C. Data Statistical Analysis · p. 12, Figure 6 · p. 13
  • Question diversity is quantified as average cosine distance between questions after punctuation removal and sentence-transformer encoding.The procedure differs from A-OKVQA, which does not remove punctuation before this calculation.Source: C. Data Statistical Analysis · p. 12

D. Data Visualization

The figures show generated instruction examples and QA pairs, including pairs generated by VIG and VIC and pairs generated by VIGC.

D. Data Visualization

  • Figure 7 presents generated instruction examples based on VIGC.Source: Figure 7 · p. 14
  • Figure 8 presents OKVQA and A-OKVQA QA pairs generated by VIG and VIC.Source: Figure 8 · p. 15
  • Figure 9 presents OKVQA and A-OKVQA QA pairs generated by VIGC.Source: Figure 9 · p. 16

Related papers

Loading 2308.12714v3…