PaperlayerView exact revision on arXiv

Source-linked AI summary

To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning

Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, Yu-Gang Jiang

arXiv:2311.07574v2cs.CV

TL;DR

Existing visual instruction tuning often relies on coarse image annotations and can produce incorrect answers without full visual context. This paper uses GPT-4V with LVIS images to build 220K visually aligned instructions, improving LLaVA-1.5 across challenging benchmarks, including LLaVA^w and MM-Vet.

  • Problem

    Language-model-generated visual instructions rely on coarse annotations and may be inaccurate without the complete visual context.

    Source: §1 · p. 1, §3 · p. 3, §5 · p. 7

  • Method

    GPT-4V generates 220K fine-grained, visually aligned instructions from 110K LVIS images and their annotations.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Replacing LLaVA-Instruct with LVIS-Instruct4V improves LLaVA-1.5 across most benchmarks, reaching 76.7 vs. 70.7 on LLaVA^w and 40.2 vs. 35.4 on MM-Vet.

    Source: §1 · p. 2, §4.2 · pp. 5–6

  • Takeaways & Limitations

    Higher-quality visual instructional data can improve large multimodal models across traditional and challenging multimodal benchmarks.

    Source: §1 · p. 2, §2.2 · p. 3, §4.2 · p. 6

Abstract

from arXiv · show

Existing visual instruction tuning methods typically prompt large language models with textual descriptions to generate instruction-following data. Despite the promising performance achieved, these descriptions are derived from image annotations, which are oftentimes coarse-grained. Furthermore, the instructions might even contradict the visual content without observing the entire visual context. To address this challenge, we introduce a fine-grained visual instruction dataset, LVIS-Instruct4V, which contains 220K visually aligned and context-aware instructions produced by prompting the powerful GPT-4V with images from LVIS. Through experimental validation and case studies, we demonstrate that high-quality visual instructional data could improve the performance of LLaVA-1.5, a state-of-the-art large multimodal model, across a wide spectrum of benchmarks by clear margins. Notably, by simply replacing the LLaVA-Instruct with our LVIS-Instruct4V, we achieve better results than LLaVA on most challenging LMM benchmarks, e.g., LLaVA$^w$ (76.7 vs. 70.7) and MM-Vet (40.2 vs. 35.4). We release our data and model at https://github.com/X2FD/LVIS-INSTRUCT4V.

1 Introduction

Visual instruction tuning commonly relies on language models prompted with image annotations, but missing visual context can omit crucial details or produce incorrect fine-grained answers. LVIS-Instruct4V addresses this by using GPT-4V with images to create 220K detailed instructions, improving LLaVA-1.5 across benchmarks.

1 Introduction

  • GPT-4 prompted without direct image access can omit crucial visual details or generate incorrect answers to fine-grained questions about color and location.Existing approaches typically generate instruction-answer pairs from captions, bounding boxes, or other textual image descriptions.Source: §1 · p. 1, Figure 1 · p. 2
  • GPT-4V was prompted with LVIS images and annotations to produce context-aware conversations and high-quality image descriptions.The method targets more accurate, context-aware, and fine-grained instruction-following data.Source: §1 · pp. 1–2
  • LVIS-Instruct4V contains 220K detailed instructions designed to align visual content with instruction-following data.Source: §1 · p. 2
  • 76.7 vs. 70.7 on LLaVA^w and 40.2 vs. 35.4 on MM-Vet were achieved by replacing LLaVA-Instruct with LVIS-Instruct4V in LLaVA-1.5.The same model architecture was used, and mixing the two instruction datasets improved results further.Source: §1 · p. 2

2 Related Work

Large multimodal models extend language-model comprehension and reasoning to visual inputs, while visual instruction tuning adapts language models to follow multimodal instructions. LVIS-Instruct4V improves this data-generation paradigm by giving GPT-4V direct access to images.

2.1 Large Multimodal Models

  • LMMs extend pretrained language models to visual tasks such as image captioning and visual question-answering.Prior systems address cross-modal alignment through architectures including BLIP-2's lightweight Q-Former.Source: §2.1 · p. 2

2.2 Visual Instruction Tuning

  • Visual instruction tuning enables language models to follow visual instructions and interact with multimodal information.LLaVA introduced this direction after instruction tuning was first developed for text-only NLP.Source: §2.2 · p. 2

2 Related Work

  • Figure 2 compares the proposed approach with other state-of-the-art methods across multiple benchmarks, with detailed numbers reported in Table 1.Source: Figure 2 · p. 3
  • LVIS-Instruct4V uses GPT-4V's multimodal understanding to generate instruction-answer pairs that are more visually aligned and richer in visual details.Direct image access during generation distinguishes the dataset from language-only instruction construction.Source: §2.2 · p. 3

3 LVIS-INSTRUCT4V

LVIS-Instruct4V conditions instruction generation on both images and language, using GPT-4V to produce conversational and descriptive data from LVIS. The resulting instructions contain richer fine-grained visual information than LLaVA-Instruct.

3 LVIS-INSTRUCT4V

  • 220K instructions were generated from 110K LVIS images, split into 110K conversational and 110K descriptional examples.GPT-4V generates question-answer lists through self-reasoning and image descriptions guided by precise bounding boxes.Source: §3 · pp. 3–4
  • Conditioning generation on visual context incorporates object categories, attributes, and interrelationships into instruction-following data.This addresses omissions and inaccuracies caused by language-only generation from annotations.Source: §3 · p. 3
  • The proposed instructions are more detailed than LLaVA's because they are generated from both visual and language inputs.Figure 3 provides qualitative examples comparing the two datasets.Source: §3 · p. 4, Figure 3 · p. 4
  • LVIS-Instruct4V contains more fine-grained information across position, count, size, color, material, and shape clues than LLaVA-Instruct.The analysis counts clue-related words and normalizes their occurrence by the total number of instructions.Source: §3 · p. 4, Figure 4 · p. 4

4 Experiment

Experiments show that replacing LLaVA-Instruct with LVIS-INSTRUCT4V improves LLaVA-1.5 across diverse benchmarks, with further gains from mixing both datasets. Ablations and qualitative comparisons attribute benefits to the dataset’s visual detail and alignment.

4.2 Comparison with State-of-the-art Methods

  • 79.2 on VQAv2 with Vicuna-7B rises to 80.1 with a 13B language model, while replacing LLaVA-Instruct improves challenging LMM benchmarks.The comparison uses the same LLaVA-1.5 architecture while changing the instruction-tuning data.Source: §4.2 · p. 5, Table 1 · p. 5
  • 9 out of 11 benchmarks favor the model additionally trained with LVIS-Instruct4V and LLaVA-Instruct, surpassing LLaVA across all benchmarks.The model achieves a 43.6-point gain on MME, from 1531.3 to 1574.9, and a 5.6% improvement on MM-Vet.Source: §4.2 · pp. 5–6

4.3 Discussion

  • Freezing the language model still improves MME and SEED over LLaVA-1.5, while fine-tuning LLaVA-1.5 raises MME from 1531.3 to 1581.3.The ablation compares optimization choices using the same instruction data.Source: §4.3 · p. 6, Table 2 · p. 6
  • Qualitative results show better recognition of small text and identification of relationships among objects after training with LVIS-INSTRUCT4V.The visual comparison is summarized as more accurate scene-text recognition and finer-grained differentiation.Source: §4.3 · p. 6, Table 3 · p. 6

5 Conclusion

The paper concludes that LVIS-INSTRUCT4V provides fine-grained, visually aligned instructions by using GPT-4V to analyze images directly. Experiments across diverse benchmarks show that this dataset benefits visual instruction tuning.

5 Conclusion

  • LVIS-INSTRUCT4V contains 220K fine-grained, visually aligned instructions curated by prompting GPT-4V with images.The dataset addresses coarse annotations and inaccurate answers produced without full visual context.Source: §5 · p. 7
  • Experiments on a wide range of benchmarks clearly demonstrate that LVIS-INSTRUCT4V benefits visual instruction tuning.The evaluation builds upon LLaVA to test the dataset’s efficacy.Source: §5 · p. 7

A Pseudo code for instruction data generation

The generation procedure prompts GPT-4V to create visual conversations and comprehensive captions from images, object annotations, and bounding boxes. It explicitly requests diverse visual questions and context-aware descriptions.

A Pseudo code for instruction data generation

  • GPT-4V is prompted to generate diverse conversations covering object types, counts, actions, locations, and other visual content.The prompt frames the model as a visual assistant answering questions while seeing the image.Source: A Pseudo code for instruction data… · p. 10
  • The captioning prompt supplies object categories and normalized bounding boxes, then requests comprehensive descriptions of each object set.Bounding boxes use normalized coordinates represented as (x1, y1, x2, y2).Source: A Pseudo code for instruction data… · p. 10

B Vocabulary List

The paper defines a vocabulary list for analyzing instruction content. The listed words target fine-grained visual attributes such as position, count, size, color, material, and shape.

B Vocabulary List

  • The vocabulary list supports instruction analysis of fine-grained visual clues, including position, count, size, color, material, and shape.These categories are used to compare word occurrence in LLaVA-Instruct and LVIS-Instruct4V.Source: B Vocabulary List · p. 11, Table 4 · p. 11

Related papers

Loading 2311.07574v2…