PaperlayerView exact revision on arXiv

Source-linked AI summary

SVIT: Scaling up Visual Instruction Tuning

Bo Zhao, Boya Wu, Muyang He, Tiejun Huang

arXiv:2307.04087v3cs.CV

TL;DR

Visual instruction tuning datasets remain small and less informative, leaving multimodal models’ capabilities under-explored. SVIT builds a 4.2-million-example dataset and trains SVIT-v1.5, which outperforms state-of-the-art MLLMs on multiple benchmarks.

  • Problem

    Existing visual instruction datasets are small and less informative, limiting evidence about the capabilities of multimodal models.

    Source: §1 · p. 1

  • Method

    SVIT builds 4.2 million instruction-tuning examples from Visual Genome annotations using GPT-4 and proposes a recipe for selecting diverse, balanced subsets.

    Source: §1 · pp. 1–2

  • Results

    SVIT-v1.5 and models trained on SVIT subsets outperform state-of-the-art MLLMs on multiple benchmarks.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The study concludes that SVIT offers a large-scale dataset and subset-selection recipe for visual instruction tuning.

    Source: §6 · p. 11

    Some answers are correct but do not address the questions precisely.

    Source: §3.4 · p. 7

Abstract

from arXiv · show

Thanks to the emerging of foundation models, the large language and vision models are integrated to acquire the multimodal ability of visual captioning, question answering, etc. Although existing multimodal models present impressive performance of visual understanding and reasoning, their limits are still largely under-explored due to the scarcity of high-quality instruction tuning data. To push the limits of multimodal capability, we Scale up Visual Instruction Tuning (SVIT) by constructing a dataset of 4.2 million visual instruction tuning data including 1.6M conversation question-answer (QA) pairs, 1.6M complex reasoning QA pairs, 1.0M referring QA pairs and 106K detailed image descriptions. Besides the volume, the proposed dataset is also featured by the high quality and rich diversity, which is generated by prompting GPT-4 with the abundant manual annotations of images. We also propose a new data recipe to select subset with better diversity and balance, which evokes model's superior capabilities. Extensive experiments verify that SVIT-v1.5, trained on the proposed dataset, outperforms state-of-the-art Multimodal Large Language Models on popular benchmarks. The data and code are publicly available at https://github.com/BAAI-DCAI/Visual-Instruction-Tuning.

1. Introduction

SVIT addresses limitations in visual instruction tuning data with a large, diverse dataset and a data-selection recipe, then trains SVIT-v1.5, which outperforms state-of-the-art MLLMs on popular benchmarks.

  • Existing visual instruction datasets are small and often focus on short captions and simple questions, while language-only GPT models struggle to imagine whole images precisely, limiting generated data’s diversity and complexity.Source: §1 · p. 1
  • The dataset combines Visual Genome’s manual annotations with GPT-4 prompts using image-level descriptions, detailed region descriptions, and object bounding boxes.Source: §1 · p. 2
  • SVIT provides 4.2M instruction-tuning examples: 1.6M conversation QA pairs, 1.6M complex reasoning QA pairs, 1.0M referring QA pairs, and 106K detailed image descriptions.Source: §1 · pp. 1–2
  • A new data recipe selects an informative subset with greater diversity and balance to better match downstream tasks.Source: §1 · p. 2
  • SVIT-v1.5 outperforms state-of-the-art MLLMs, including LLaVA-v1.5, Qwen-VL-Chat, and InstructBLIP, on popular benchmarks.Source: §1 · p. 2

2. Related Work

The review situates SVIT among end-to-end multimodal models and visual-instruction datasets, highlighting prior model designs, limits in existing visual instruction data, and instruction data for other modalities.

2.1. Multimodal Models

2.2. Multimodal Instruction Tuning

  • Visual instruction tuning includes refined image-text pairs, GPT-4-generated LLaVA-Instruct-150K, OCR/GPT-4 LLaVAR, and unified M3IT, yet public datasets often remain short, perception-focused QA.Zhu et al. (2023a) refine 3,500 pairs; Dai et al. (2023) combine 26 public datasets, while Li et al. (2023d) convert 40 datasets and paraphrase answers.Source: §2.2 · p. 3
  • SVIT uses rich image- and region-level annotations from Visual Genome and COCO to generate 4.2M visual instruction data spanning perception, reasoning, and planning.Source: §2.2 · p. 3
  • Other multimodal instruction data target video, RGB-D, speech, and audio; EgoCOT uses video captions for embodied-planning instructions, while MIMIC-IT draws largely on egocentric E4D videos.Source: §2.2 · p. 3

3. Dataset Construction

SVIT builds visual instruction data from dense image annotations, uses GPT-4 to generate four instruction tasks, filters outputs, and analyzes the resulting dataset’s scale, question mix, and errors.

3.1. Source Data

  • SVIT draws on 257,633 COCO captions, 3,802,374 Visual Genome object names with boxes, and 5,406,592 region descriptions with boxes.Visual Genome has 108,077 densely annotated images, averaging 42 human-generated region descriptions and 21 objects per image; it is partially sourced from COCO.Source: §3.1 · p. 3

3.2. Instruction Data Generation

  • GPT-4 generates conversations, complex-reasoning QAs, referring QAs, and detailed image descriptions from prompts, without examples, to encourage diversity.The design draws inspiration from LLaVA (Liu et al., 2023b): each image receives three five-pair conversations, 15 complex-reasoning QAs, and 10 region-specific QAs; referring objects are marked with normalized bounding boxes.Source: §3.2 · pp. 3, 5

3.3. Postprocessing

  • The postprocessing pass regenerates answers containing unnecessary content and replaces conversations or QA sets that fall short of the required number until they satisfy the filters.Source: §3.3 · p. 5

3.4. Statistics and Analysis

  • SVIT contains 1,565,797 conversation QAs, 1,556,902 complex reasoning QAs, 1,011,338 referring QAs, and 106,274 detailed image descriptions.Detailed descriptions average 361.5 words, compared with 11.3 words for COCO captions; conversation, complex reasoning, and referring answers average 27.9, 26.6, and 20.6 words, respectively.Source: §3.4 · p. 5
  • “What” questions are the largest category in conversation (38%), complex reasoning (55%), and referring QAs (41%), while complex reasoning includes 9% “why” and 11% “how” questions.Only 11% of SVIT’s “how” questions are “how many” questions; referring QAs also cover positions, object existence, suggestions, planning, reasoning, and distinguishing objects.Source: §3.4 · p. 5, Figure 3 · p. 6
  • Manual review found around 5% of questions could receive more accurate or satisfying answers, due to annotation errors, imprecise answers, or incorrect generated content.Examples include incorrect COCO captions or Visual Genome object labels, answers that fail to address a question precisely, and image descriptions that misplace objects.Source: §3.4 · pp. 5, 7

4. Method

The method adapts LLaVA through staged visual-instruction training and selects benchmark-oriented coreset data for diversity and balance.

4.1. Model Architecture

  • LLaVA (Liu et al., 2023b;a) encodes images, converts features into language tokens, and combines them with instruction tokens for response generation.Its architecture comprises a vision encoder, vision-language connector, and large language model.Source: §4.1 · p. 7
  • Training first updates only the connector on image-text pairs, then updates the connector and LLM parameters through full-parameter or LoRA tuning (Hu et al., 2022).The vision encoder and LLM stay frozen during pre-training; in LoRA fine-tuning, learnable LoRA parameters are updated instead of full LLM parameters.Source: §4.1 · p. 7

4.2. Coreset Selection Algorithm

  • The coreset recipe selects informative samples by overlap with benchmark-related concepts and re-samples data to balance “Yes” and “No” questions.GPT-4 generates keywords for high-level concepts matching MME (Fu et al., 2023) and MMBench (Liu et al., 2023c); low-frequency keywords in SVIT are filtered out.Source: §4.2 · p. 7
  • The resulting SVITcore-150K contains 157,712 samples, matching LLaVA-Instruct-150K in size, and SVIT-mix-665K replaces that set in LLaVA-v1.5-mix-665K.Source: §4.2 · p. 7

5. Experiments

SVIT-v1.5 outperforms LLaVA-v1.5 and other MLLMs in most benchmark settings; LLaVA-v1.0 MME ablations assess data quality, diversity, balance, and scale, while qualitative examples demonstrate model abilities.

5.1. Comparison to the State of the Art

  • SVIT-v1.5 (Full) scores 1565.8 on MME perception, 34.5 above LLaVA-v1.5 (Full); SVIT-v1.5 (LoRA) scores 364.3 vs. 300.4 in MME cognition (+63.9).The authors attribute the improvements to SVIT data, with the same data amount and base model used for comparison.Source: Table 2 · p. 8, §5.1 · p. 8

5.2. Ablation Study

  • SVIT-80K improves MME performance by 8.2% over LLaVA-Instruct-80K, using an equally sized random subset of 20K examples from each SVIT category.The authors attribute its gains to detailed image annotations and prompts designed to cover a wide range of tasks; reported perception gains include count (+109.1%) and posters (+23.1%), while cognition gains include numerical calculation (+27.8%).Source: §5.2 · pp. 8–9
  • SVIT-80K-D, selected by removing less-informative samples and then sampling 20K examples per category, improves MME by 20.3 score over random SVIT-80K.Source: §5.2 · p. 9
  • With a Yes:No training ratio of 8, SVIT-80K-B improves MME by 7.1% over SVIT-80K and produces more uniform Yes/No responses.A balanced ratio of 1 instead led the model to respond “No” for almost any question; the ratio-8 recipe brought responses closer to MME’s 1:1 Yes/No prior.Source: Figure 5 · p. 9, §5.2 · pp. 9–10

5.3. Qualitative Evaluation

  • Qualitative examples show SVIT-v1.5 describing scenes and offering suggestions, planning next steps, locating objects with bounding boxes, distinguishing real from synthetic images, and interpreting comics.Source: Figure 6 · p. 10, §5.3 · p. 11

5.2. Ablation Study

  • On MME, fine-tuning with SVIT-train, compared with randomly selected SVIT-80K without a data recipe, improves total score by 12.7%, including object-existence (+105.6%), object-color (+100.0%), and OCR (+39.1%) gains.Source: §5.2 · p. 10

6. Conclusion

SVIT scales visual instruction tuning with a 4.2-million-example dataset and a sample-selection recipe designed for better diversity and balance. Experiments report that SVIT-v1.5 outperforms state-of-the-art MLLMs on multiple benchmarks.

  • Experiments report that SVIT-v1.5, trained on the proposed dataset and its subsets, outperforms state-of-the-art MLLMs on multiple benchmarks.Source: §6 · p. 11
  • SVIT provides a large-scale visual instruction-tuning dataset containing 4.2 million examples.Source: §6 · p. 11
  • The proposed sample-selection recipe aims to improve the diversity and balance of the data.Source: §6 · p. 11

Appendix · A. Prompts

The appendix describes four GPT-4 prompt tasks built from image captions, object bounding boxes, and region descriptions, with input and output constraints varying by task.

A. Prompts

  • Four tasks are designed to prompt GPT-4 based on image captions, object bounding boxes, and region descriptions.Source: A. Prompts · p. 15
  • Each prompt begins with a shared description of the input data, then differs by task description, summarized in Figure 7.Source: A. Prompts · p. 15
  • Conversation, complex reasoning, and detail-description inputs omit region-description bounding boxes because they may exceed GPT-4’s context limit.Source: A. Prompts · p. 15
  • Referring-QA inputs include region-description bounding boxes because location information is vital to accurate image understanding.Source: A. Prompts · p. 15
  • Referring-QA responses are shortened to 10 QAs per image to fit the context limit.Source: A. Prompts · p. 15
  • The referring-QA prompt is summarized in Figure 8.Source: A. Prompts · p. 15

B. Instructions for Detail Description

Detail-description instructions are generated by prompting GPT-4 for varied phrasings and randomly sampling one instruction for each image.

B. Instructions for Detail Description

C. Concept Set

The concept set measures training-sample informativeness using keywords generated by GPT-4 and filtered by their frequency in the dataset.

C. Concept Set

  • The concept set measures training-sample informativeness with GPT-4-generated keywords filtered by their frequency in the dataset.Table 4 presents the concepts and their keywords.Source: C. Concept Set · p. 15, Table 4 · p. 15

Instructions for detail description

Figure 9 presents instructions for detail description.

Instructions for detail description

  • Figure 9 is titled “Instructions for detail description.”Source: Figure 9 · p. 18

Related papers

Loading 2307.04087v3…