PaperlayerView exact revision on arXiv

Source-linked AI summary

SVIT: Scaling up Visual Instruction Tuning

Bo Zhao, Boya Wu, Muyang He, Tiejun Huang

arXiv:2307.04087v3cs.CV

TL;DR

Existing multimodal models are limited by the scarcity of high-quality visual instruction data. SVIT scales visual instruction tuning with a 4.2M-example dataset and a diversity-and-balance selection recipe, and SVIT-v1.5 outperforms state-of-the-art multimodal models on multiple benchmarks. The data also contains some incorrect annotations and answers that are correct but imprecise.

  • Problem

    Existing visual instruction datasets are small and less informative because high-quality data is costly to construct.

  • Method

    SVIT constructs 4.2M visual instruction examples from image annotations and GPT-4 prompting, then selects diverse and balanced subsets for model fine-tuning.

  • Results

    SVIT-v1.5 trained on the proposed dataset and subsets outperforms state-of-the-art multimodal large language models on multiple benchmarks.

  • Takeaways & Limitations

    Scaling visual instruction tuning with diverse, balanced data improves the reported capabilities and benchmark performance of SVIT-v1.5.

  • Takeaways & Limitations

    The constructed data includes incorrect annotations and answers that may be correct but fail to address questions precisely.

Abstract

from arXiv · show

Thanks to the emerging of foundation models, the large language and vision models are integrated to acquire the multimodal ability of visual captioning, question answering, etc. Although existing multimodal models present impressive performance of visual understanding and reasoning, their limits are still largely under-explored due to the scarcity of high-quality instruction tuning data. To push the limits of multimodal capability, we Scale up Visual Instruction Tuning (SVIT) by constructing a dataset of 4.2 million visual instruction tuning data including 1.6M conversation question-answer (QA) pairs, 1.6M complex reasoning QA pairs, 1.0M referring QA pairs and 106K detailed image descriptions. Besides the volume, the proposed dataset is also featured by the high quality and rich diversity, which is generated by prompting GPT-4 with the abundant manual annotations of images. We also propose a new data recipe to select subset with better diversity and balance, which evokes model's superior capabilities. Extensive experiments verify that SVIT-v1.5, trained on the proposed dataset, outperforms state-of-the-art Multimodal Large Language Models on popular benchmarks. The data and code are publicly available at https://github.com/BAAI-DCAI/Visual-Instruction-Tuning.

1. Introduction

SVIT addresses the limited scale and informativeness of existing visual instruction datasets by constructing a diverse 4.2M-example dataset and a balanced subset-selection recipe. The resulting SVIT-v1.5 model is reported to outperform state-of-the-art multimodal large language models on popular benchmarks.

  • Existing visual instruction datasets remain small and less informative because constructing high-quality data is costly.
  • 4.2M examples comprise 1.6M conversation QA pairs, 1.6M complex reasoning QA pairs, 1.0M referring QA pairs, and 106K detailed image descriptions.
  • SVIT uses Visual Genome annotations and GPT-4 prompting to enrich instruction-tuning data with greater diversity and informativeness.The prompts use image-level descriptions, region descriptions, and object bounding boxes.
  • The proposed data recipe selects an informative subset with better diversity and balance for downstream tasks.
  • SVIT-v1.5 outperforms state-of-the-art multimodal large language models on popular benchmarks.

2. Related Work

Related work spans multimodal systems, end-to-end vision-language models, and instruction-data construction across images, videos, RGB-D data, speech, and audio. Prior studies emphasize that high-quality visual instruction data is important for multimodal model performance.

  • Multimodal solutions include coordinated systems with language and vision models and end-to-end differentiable models that fuse pretrained components.
  • End-to-end models use mechanisms such as gated cross-attention, Q-Former modules, or linear layers to connect visual and language representations.
  • Table 1 compares SVIT with similar vision-language instruction datasets generated by GPT.
  • Prior work reports that high-quality image-text data for fine-tuning is important to multimodal model performance, while short-answer VQA-derived data can degrade performance.
  • Instruction data has also been developed for videos, RGB-D images, speech, and audio, including data generated by prompting ChatGPT with visual inputs or captions.

3. Dataset Construction

SVIT constructs visual instruction data by prompting GPT-4 with dense Visual Genome and COCO annotations across four task types, then filters and analyzes the generated content. The resulting collection contains over 4.2 million examples with varied question forms, while manual inspection identifies annotation and answer-quality errors.

  • Data Sources: Visual Genome provides 108,077 densely annotated images, while COCO contributes high-level captions to complement region-level descriptions.The annotations include regions, objects, attributes, and relationships.
  • Instruction Data Generation: GPT-4 generates conversations, complex-reasoning QAs, referring QAs with normalized bounding boxes, and detailed image descriptions.The four tasks target connected dialogue, step-by-step reasoning, region grounding, and rich visual description.
  • Quality Control: GPT-4 outputs are regenerated when they contain unnecessary references to source descriptions or fail to meet the required number of responses.The filtering and regeneration procedure is repeated until the content satisfies the requirements.
  • Statistics: 1,565,797 conversation QAs, 1,556,902 complex reasoning QAs, 1,011,338 referring QAs, and 106,274 detailed descriptions are obtained through a two-pass procedure.Average question-answer lengths exceed those in the original Visual Genome data.
  • Question Distribution: “What” questions form the largest category in conversations (38%), complex reasoning (55%), and referring QAs (41%).Other recurring forms include where, is/are there any, what suggestion, why, and which questions.
  • Quality Analysis: Manual examination finds that around 5% of questions could receive a more accurate or satisfying answer, including annotation errors, imprecise answers, and incorrect descriptions.Examples include wrong object labels, incorrect counts, answers that do not precisely address questions, and misplaced objects.

4. Method

The method uses a two-stage LLaVA-based training pipeline and a benchmark-oriented coreset selection recipe. The recipe targets informative, diverse, and balanced visual instruction data while trading off benchmark performance and training efficiency.

  • Model architecture: LLaVA combines a vision encoder, vision-language connector, and language model to convert image features into language tokens for response generation.The connector maps extracted image features into language embedding tokens before the combined vision and language tokens enter the language model.
  • Training procedure: Training proceeds in two stages: connector pre-training on image-text pairs, followed by visual-instruction fine-tuning.During pre-training, the vision encoder and language-model weights remain frozen; fine-tuning uses full-parameter or LoRA tuning.
  • Coreset selection: The coreset selection algorithm targets benchmark adaptation by balancing performance and training efficiency.It selects a subset of instruction data rather than using the complete dataset.
  • Coreset selection: Diversity selection measures each sample's overlap with benchmark-matched concepts and keeps the most informative samples.The concept set is built from high-level concepts and GPT-4-generated keywords associated with MME and MMBench.
  • Coreset selection: Balance selection re-samples Yes/No examples because GPT-4-generated data contains an extremely imbalanced proportion of the two choices.The authors study how the Yes:No proportion relates to MME performance.

5. Experiments

Experiments compare SVIT-v1.5 with state-of-the-art models and evaluate data quality, diversity, balance, and scaling strategies. Across these evaluations, SVIT-based training improves benchmark performance, including gains from larger, more diverse, and better-balanced subsets.

  • 5.1. Comparison to the State of the Art: SVIT-v1.5 outperforms LLaVA-v1.5 and other models in most settings across 11 multimodal benchmarks.The comparison uses the LLaVA-v1.5-13B architecture and the SVIT-mix-665K training set.
  • 5.1. Comparison to the State of the Art: 1565.8 score in MME perception exceeds LLaVA-v1.5 by 34.5 score for full-parameter training.With LoRA training, SVIT-v1.5 exceeds LLaVA-v1.5 by 63.9 score in MME cognition, 364.3 versus 300.4.
  • 5.2. Ablation Study: +8.2% performance is achieved by SVIT-80K over LLaVA-Instruct-80K on the MME benchmark at the same data scale.SVIT-80K particularly improves count, posters, scene, landmark, artwork, commonsense reasoning, and numerical calculation tasks.
  • 5.2. Ablation Study: 20.3 score improvement over random SVIT-80K selection is obtained by the diversity-enhanced SVIT-80K-D subset.The subset removes less-informative samples and samples 20K data from each SVIT category.
  • 5.2. Ablation Study: 7.1% improvement over SVIT-80K is achieved by SVIT-80K-B using a Y:N ratio of 8, producing more uniform Yes/No responses.A balanced training ratio of 1 instead made the model very likely to respond “No” to questions.
  • 5.2. Ablation Study: +12.7% score improvement on MME is achieved by SVIT-train over randomly selected SVIT-80K.Larger training data particularly improves object existence, object color, and OCR abilities.

6. Conclusion

The paper scales visual instruction tuning with the 4.2-million-example SVIT dataset and introduces subset-selection recipes for diversity and balance. Experiments report that SVIT-v1.5 and its subsets outperform state-of-the-art multimodal large language models on multiple benchmarks.

  • 6. Conclusion: SVIT contains 4.2 million instruction-tuning examples, while its selection recipes target better diversity and balance.The recipes support training SVIT-v1.5 and subsets evaluated against state-of-the-art multimodal large language models.

A. Prompts

The prompts use image captions, object bounding boxes, and region descriptions to generate four visual instruction-tuning tasks with GPT-4. Inputs omit region descriptions and bounding boxes for several tasks when context length would otherwise exceed GPT-4’s limit.

  • A. Prompts: Four GPT-4-prompted tasks are designed from image captions, object bounding boxes, and region descriptions.Bounding boxes and region descriptions are excluded from conversation, complex reasoning, and detail description inputs when context length may exceed GPT-4’s limit.

B. Instructions for Detail Description

For detail description, GPT-4 receives varied phrasings of the request to describe an image in detail. One instruction is randomly sampled for each image.

  • B. Instructions for Detail Description: One instruction is randomly sampled per image from multiple phrasings of “can you describe the image in detail.”The accumulated alternatives are used to vary detail-description prompts.

C. Concept Set

The paper defines a concept set to measure training-sample informativeness and uses GPT-4 prompts to generate several types of visual instruction data. The prompting setup covers conversation, complex reasoning, detail description, and referring question answering.

  • C. Concept Set: The concept set measures the informativeness of training samples using GPT-4-generated keywords filtered by dataset frequency.The concept set and its keywords are presented in Table 4.
  • C. Concept Set: The visual assistant prompt represents image information through captions and object descriptions with normalized bounding boxes.The object representation includes an object name and coordinates in the form [x1, y1, x2, y2].
  • C. Concept Set: GPT-4 is prompted to generate conversation, complex reasoning, and detailed image-description data.Figure 7 presents the prompts used for these three instruction types.
  • C. Concept Set: Referring question-answer data is generated with a dedicated GPT-4 prompt for analyzing a single image.The referring-QA instructions describe the image through captions and other structured information.

Instructions for detail description

The detail-description instruction asks GPT-4 to provide a comprehensive description of an image. The corresponding figure documents the instructions used for this task.

  • Instructions for detail description: The instruction requests a comprehensive description of the image.
  • Instructions for detail description: The task focuses on producing detailed image descriptions through an instruction-driven prompting process.
  • Instructions for detail description: The detail-description task is specified through dedicated instructions shown in Figure 9.
Loading 2307.04087v3…