PaperlayerView exact revision on arXiv

Source-linked AI summary

COCO is "ALL'' You Need for Visual Instruction Fine-tuning

Xiaotian Han, Yiqi Wang, Bohan Zhai, Quanzeng You, Hongxia Yang

arXiv:2401.08968v1cs.CV

TL;DR

Visual IFT is intended to align MLLMs with user instructions, but multi-round dialogue and open-ended evaluation remain concerns for current systems and benchmarks. The paper builds an IFT dataset by merging diverse annotations for COCO images and evaluates dialogue across rounds. The authors report better instruction following and equal or better open-ended benchmark performance with their dataset.

  • Problem

    Current IFT models may fail to follow instructions in multi-round conversations, while caption and VQA benchmarks with closed-form metrics are not well suited to generative MLLMs.

    Source: §1 · p. 3

  • Method

    The paper builds a COCO-image-centric IFT dataset by merging annotations from multiple sources into multi-round conversations with varied instructions.

    Source: §1 · p. 3, §3.1 · p. 4, §3.2 · p. 4

  • Results

    The authors report better instruction following and equal or better open-ended benchmark performance for models trained with their dataset.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The authors conclude that COCO-based merged annotations can support visual IFT and call for more comprehensive study of dataset construction and open-ended MLLM evaluation.

    Source: §5 · p. 7

    Annotations are merged into multi-round conversations without accounting for their order or logical relationship, and samples interleaving multiple images with text are rare.

    Source: §5 · p. 7

Abstract

from arXiv · show

Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. Visual instruction fine-tuning (IFT) is a vital process for aligning MLLMs' output with user's intentions. High-quality and diversified instruction following data is the key to this fine-tuning process. Recent studies propose to construct visual IFT datasets through a multifaceted approach: transforming existing datasets with rule-based templates, employing GPT-4 for rewriting annotations, and utilizing GPT-4V for visual dataset pseudo-labeling. LLaVA-1.5 adopted similar approach and construct LLaVA-mix-665k, which is one of the simplest, most widely used, yet most effective IFT datasets today. Notably, when properly fine-tuned with this dataset, MLLMs can achieve state-of-the-art performance on several benchmarks. However, we noticed that models trained with this dataset often struggle to follow user instructions properly in multi-round dialog. In addition, tradition caption and VQA evaluation benchmarks, with their closed-form evaluation structure, are not fully equipped to assess the capabilities of modern open-ended generative MLLMs. This problem is not unique to the LLaVA-mix-665k dataset, but may be a potential issue in all IFT datasets constructed from image captioning or VQA sources, though the extent of this issue may vary. We argue that datasets with diverse and high-quality detailed instruction following annotations are essential and adequate for MLLMs IFT. In this work, we establish a new IFT dataset, with images sourced from the COCO dataset along with more diverse instructions. Our experiments show that when fine-tuned with out proposed dataset, MLLMs achieve better performance on open-ended evaluation benchmarks in both single-round and multi-round dialog setting.

1 Introduction

Visual IFT aligns pretrained MLLMs with user intent, but existing evaluations can miss open-ended capabilities and models may fail to follow instructions across dialogue rounds. The paper introduces a COCO-image-centric IFT dataset and a multi-round evaluation protocol.

  • Traditional caption and VQA benchmarks rely on short outputs and closed-form metrics, limiting their fit for evaluating generative MLLMs.Source: §1 · pp. 1, 3
  • The authors identify multi-round instruction following as a weakness, with LLaVA-1.5 sometimes giving one-word answers despite requests for longer responses.They frame IFT as aligning with user intent, rather than injecting knowledge.Source: §1 · p. 3
  • The paper constructs a COCO-image-centric visual IFT dataset by merging annotations from multiple IFT dataset sources.Source: §1 · p. 3
  • The proposed dataset is reported to outperform official LLaVA-1.5-13B on open-ended evaluation benchmarks.Source: §1 · p. 3
  • The authors design a protocol to evaluate MLLMs in multi-round dialogue.Source: §1 · p. 3
  • The paper claims that a few images with high-quality instruction-following annotations suffice for IFT, while adding more GQA or VQA data leads to in-domain overfitting.Source: §1 · p. 3

2 Background

The background distinguishes end-to-end MLLMs from tool-using systems and describes the stages and common data-construction approaches for visual instruction fine-tuning.

2.1 Multi-modal Large Language Models

  • End-to-end MLLMs combine a pretrained visual encoder, a pretrained LLM, and a vision-language bridge; the paper focuses on this type rather than tool-using systems.Source: §2.1 · p. 3

2.2 Visual Instruction Fine-Tuning

  • Visual IFT tunes models on visual data with text or visual instructions, using templates, GPT-4-generated conversations, or GPT-4V-generated captions and questions.Some works add continual pretraining between pretraining and IFT to inject visual knowledge.Source: §2.2 · p. 3

3 Dataset

The dataset combines annotations associated with COCO and Visual Genome images, then converts and merges them into multi-round conversational training samples.

3.1 Data Source

  • COCO and Visual Genome share about half their training images, and their annotations span image captions, visual question answering, and conversations.The two training splits contain over 150k images; the paper describes the derived datasets as covering different image aspects.Source: §3.1 · p. 4

3.2 Instruction Generation

  • The authors merge datasets by COCO image index so each image can have multi-round conversations spanning different content aspects and description levels.They preserve full-sentence QA answers and include explanations when available, producing 118,000 samples.Source: §3.2 · pp. 4–5

4 Experiments

The experiments test whether a COCO-centric IFT dataset can sustain open-ended performance in multi-round dialogue, against the observed weakness of LLaVA-1.5. Under the proposed protocol, the COCO-trained model remains similar across rounds, while original LLaVA-1.5 drops.

4 Experiments

  • The authors attribute multi-round weakness to LLaVA-mix-665k's over 300,000 short-response GQA and Visual Genome samples, hypothesizing these may hurt open-ended or multi-round ability without careful conversion.Source: §4 · pp. 5–6
  • The protocol primes each dialogue with a fixed color question and scores only the second-round answer to an annotated image question.The same pretrained LLaVA-1.5-13B checkpoint was fine-tuned for the comparison.Source: §4 · p. 6
  • Across MM-Vet and InfiMM-Eval, COCO-centric IFT shows similar single- and multi-round performance, while original LLaVA-1.5 drops significantly under multi-round evaluation.The largest MM-Vet drops are in Generation and Knowledge, and the largest InfiMM-Eval drop is in Abductive Reasoning.Source: Table 3 · p. 6, Table 4 · p. 6, §4 · pp. 6–7

5 Conclusion and Limitations

The conclusion argues that merging datasets with COCO images can support visual IFT, reporting better instruction following and equal-or-better open-ended benchmark performance. It also identifies annotation ordering, multi-image dialogue, and benchmark coverage as limitations.

5 Conclusion and Limitations

  • The authors merge datasets containing COCO images into an IFT dataset and report better instruction following with equal-or-better open-ended benchmark performance.They conclude that COCO is “all” you need for visual IFT and call for more comprehensive research on dataset construction and evaluation.Source: §5 · p. 7
  • The dataset merges same-image annotations without order or logical relations, has few interleaved multi-image samples, and lacks specialized multi-round open-ended benchmarks.Source: §5 · p. 7

Related papers

Loading 2401.08968v1…