PaperlayerView exact revision on arXiv

Source-linked AI summary

M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning

Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, Qi Liu

arXiv:2306.04387v2cs.CVcs.CL

TL;DR

Open vision-language models have lacked instruction datasets with broad task and language coverage. M3IT provides a large, multilingual collection of curated tasks and instructions, and Ying-VLM experiments report knowledgeable-VQA performance and transfer to unseen video and Chinese tasks.

  • Problem

    Open VLM instruction data is often unavailable or limited in task and language coverage, leaving a need for comprehensive multimodal instruction datasets.

    Source: §1 · p. 1

  • Method

    M3IT recasts vision-language data into a vision-to-text dataset with 40 tasks, 400 manually written instructions, 2.4 million instances, and key tasks translated into 80 languages.

    Source: §1 · p. 2, §3.2 · p. 3, §5 · p. 9

  • Results

    Ying-VLM surpasses strong baselines on knowledgeable VQA and shows strong generalization to unseen video and Chinese tasks.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The authors offer M3IT as an open dataset for developing general-purpose multimodal agents and report findings intended to support future research.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Instruction tuning has significantly advanced large language models (LLMs) such as ChatGPT, enabling them to align with human instructions across diverse tasks. However, progress in open vision-language models (VLMs) has been limited due to the scarcity of high-quality instruction datasets. To tackle this challenge and promote research in the vision-language field, we introduce the Multi-Modal, Multilingual Instruction Tuning (M$^3$IT) dataset, designed to optimize VLM alignment with human instructions. Our M$^3$IT dataset comprises 40 carefully curated datasets, including 2.4 million instances and 400 manually written task instructions, reformatted into a vision-to-text structure. Key tasks are translated into 80 languages with an advanced translation system, ensuring broader accessibility. M$^3$IT surpasses previous datasets regarding task coverage, instruction number and instance scale. Moreover, we develop Ying-VLM, a VLM model trained on our M$^3$IT dataset, showcasing its potential to answer complex questions requiring world knowledge, generalize to unseen video tasks, and comprehend unseen instructions in Chinese. We have open-sourced the dataset to encourage further research.

1 Introduction

Open vision-language models lack publicly available instruction data with broad task and language coverage. M3IT addresses this gap with a large multilingual dataset and Ying-VLM experiments that show gains on knowledgeable VQA and transfer to unseen tasks.

1 Introduction

  • Open VLM instruction data is often unavailable or limited in task and language coverage, constraining progress on comprehensive vision-language instruction tuning.The introduction identifies this scarcity as a reason high-quality multimodal instruction datasets are needed.Source: §1 · p. 1
  • M3IT unifies 40 diverse tasks and 400 manually written instructions, with key vision-language tasks translated into 80 languages.Its coverage includes image, video, and Chinese vision-language tasks.Source: §1 · p. 2
  • Ying-VLM surpasses strong baselines on knowledgeable VQA and improves generalization to unseen video and cross-lingual tasks.The model combines BLIP-2 with Ziya-13B and is trained in two stages, first aligning vision features through image captioning and then instruction-tuning on selected M3IT tasks.Source: §1 · p. 2
  • The paper presents M3IT as an open-source dataset and Ying-VLM as a visual assistant for knowledgeable VQA and generalization to unseen video and Chinese tasks.The authors frame these contributions as support for developing general-purpose multimodal agents and future research.Source: §1 · p. 2

2 Related Work

Prior multimodal instruction-tuning datasets vary in approach, but the paper identifies gaps in task coverage, instance diversity and quality, and language inclusion. M3IT is presented as an expanded dataset intended to address these gaps.

2 Related Work

  • Earlier VLM instruction-tuning efforts include generated visual-dialog data, reformatted classification tasks, and a collection of 28 image-to-text tasks.The paper names LLaVA, MiniGPT-4, MultiInstruct, and InstructBLIP as examples.Source: §2 · p. 2
  • The paper identifies limited task coverage, instance diversity and quality, and multilingual inclusion as shortcomings of existing datasets.Table 1 compares characteristics of existing datasets and M3IT, but the supplied caption does not give table values.Source: §2 · p. 2, Table 1 · p. 2
  • M3IT expands coverage to 40 datasets, adds manually written task instructions, and includes tasks in different languages.The related-work discussion presents these as responses to the identified dataset gaps.Source: §2 · p. 2

3 M3IT: A Multi-Modal Multilingual Instruction Tuning Dataset

M3IT combines broad vision-language and video-language task coverage with a staged annotation process, multilingual translation, and a unified instance format. The resulting benchmark organizes task-specific instructions, inputs, outputs, and metadata across more than 2.4 million instances.

3.1 Task Coverage

  • The task collection includes image-question answering, candidate-label classification, and visual-conditional generation such as storytelling, dialog, and multimodal translation.Some classification datasets reduce the number of candidate options to fit language-model input limits.Source: §3.1 · p. 3
  • Additional coverage includes knowledge-based VQA, Chinese vision-language tasks, and video captioning, question answering, and action classification.Knowledge-based VQA draws on outside knowledge, while the Chinese tasks include VQA, captioning, classification, and generation.Source: §3.1 · p. 3
  • M3IT spans classical vision-language tasks, from captioning and visual question answering to reasoning and classification.Its coverage includes generic and text-rich image captioning, as well as spatial, commonsense, and fine-grained semantic reasoning.Source: §3.1 · p. 3

3.2 Annotation Process

  • The annotation pipeline has four stages: writing task instructions, unifying image-text formats, checking quality, and building multilingual sets.Eight graduate-student authors familiar with the literature served as human annotators.Source: §3.2 · p. 3
  • Annotators wrote 10 diverse instructions per task, yielding 400 instructions averaging 24.4 words each.The reported average edit distance within a task is 76.6, which the authors use to indicate instruction diversity.Source: §3.2 · p. 4
  • For region-based examples, images receive red bounding boxes; short answers are paraphrased, and reviewed examples undergo format and quality checks.The review sampled 10 examples from each split; fewer than 3% of examined paraphrases were ineffective and were filtered or replaced with a template.Source: §3.2 · pp. 4–5, Figure 2 · p. 5
  • Selected task evaluation data were translated with NLLB-1.3B, retaining 80 languages after filtering by translation BLEU scores above 20.The first version translated 500 samples per split for each selected task; the selected datasets cover multiple task types.Source: §3.2 · p. 5

3.3 Dataset Format

  • Each instance has five fields—image, instruction, task-specific inputs, outputs, and metadata—and the task statistics total 2,429,264 instances.The input field can be empty for tasks such as captioning; metadata preserves information such as the original image ID.Source: §3.3 · p. 5, Table 3 · p. 6

4 Experiments

The experiments evaluate Ying-VLM on knowledgeable VQA, unseen Chinese and video tasks, and qualitative cases, alongside analyses of task count and instruction diversity. Results include gains over baselines and transfer to tasks not represented in training inputs.

4.1 Experimental Settings

  • The setup tests held-out knowledge, Chinese, and video transfer, uses ROUGE-L plus GPT-4 ratings, and compares Ying-VLM with BLIP-2-Flan-T5-XXL, MiniGPT-4, and InstructBLIP.The model uses two-stage training: visual-text alignment followed by multimodal instruction tuning.Source: §4.1 · pp. 6–7

4.2 Main Results

  • Ying-VLM improves on the strongest baseline by 3.2 ROUGE-L points on OK-VQA and 2.7 on A-OKVQA, and performs best on held-out ViQuAE.The table caption states that Ying-VLM consistently outperforms all baselines on KVQA tasks.Source: §4.2 · p. 7, Table 4 · p. 7
  • On three unseen Chinese vision-language tasks, Ying-VLM outperforms MiniGPT4 and InstructBLIP, while BLIP-2 is excluded because Flan-T5 does not support Chinese.The table caption specifies Chinese captioning, VQA, and classification tasks.Source: §4.2 · p. 7, Table 5 · p. 7
  • Ying-VLM surpasses BLIP-series baselines on video-language tasks despite training without video inputs.Evaluation uniformly samples eight frames per video; MiniGPT4 is excluded because it does not support video inputs.Source: §4.2 · pp. 7–8, Table 6 · p. 7
  • GPT-4 preferred Ying-VLM over MiniGPT4 in 167 of 300 evaluated samples, and Figure 4 reports winning rates of 55.6% against MiniGPT-4 and 65.5% against InstructBLIP.The evaluation sampled 300 examples from OK-VQA, A-OKVQA, and ViQuAE.Source: §4.2 · p. 8, Figure 4 · p. 8

4.3 Analysis

  • Increasing the number of training tasks improves generalization, with gains not diminishing across the tested task counts up to all 27 tasks.The analysis sampled 0, 4, 8, 16, and all 27 tasks, capped each task at 5,000 examples, and trained each model for 5,000 steps.Source: Figure 5 · p. 8, §4.3 · p. 9
  • Performance varies with instruction diversity, and four instructions per task are reported as sufficient for decent performance.The analysis compares one, two, four, and eight instructions per dataset.Source: Figure 6 · p. 8, §4.3 · p. 9
  • Held-out ViQuAE and ChineseFoodNet cases show natural, informative entity-centric answers and successful Chinese food classification; MiniGPT4 gives an incorrect stadium answer and generic descriptions in later cases.In the two questions requiring external knowledge, InstructBLIP answers concisely but less engagingly than Ying-VLM.Source: §4.3 · p. 9, Figure 7 · p. 9

5 Conclusion

The conclusion presents M3IT as a large multimodal instruction-tuning dataset and Ying-VLM as a validation model. The reported findings include instruction following and generalization to unseen video and Chinese tasks.

5 Conclusion

  • M3IT contains 2.4 million curated instances and 400 manually written task instructions across 40 tasks.The dataset is presented as supporting development of multimodal large language models.Source: §5 · p. 9
  • Models trained on the dataset follow human instructions, provide more engaging responses, and generalize to unseen video and Chinese tasks.The paper also reports that more tasks continually boost performance and instruction diversity influences results.Source: §5 · pp. 9–10

A Dataset Statistics

The appendix points readers to detailed benchmark statistics and flags a license-checking caveat for some datasets.

A Dataset Statistics

  • Table 7 lists detailed task descriptions and statistics for benchmark datasets, including whether each dataset is used during instruction tuning.Its “Used” column marks whether a dataset is included in that training stage.Source: Table 7 · p. 10
  • Dataset licenses were collected from PaperWith-Code, but users are advised to check project pages or contact owners for datasets with Unknown or Custom licenses.The passage identifies those license categories as requiring additional checks before use.Source: A Dataset Statistics · p. 10

B Template for Answer Paraphrase

The appendix describes a ChatGPT paraphrasing template that supplies question-answer fields and adds caption context for VQAv2.

B Template for Answer Paraphrase

C Dataset Translation

The translated datasets broaden evaluation access across languages, with translation coverage limited by available computational resources.

C Dataset Translation

  • Task instructions and evaluation sets for ImageNet, Winoground, VQAv2, OKVQA, VIST, MSRVTT, and MSRVTT-QA were translated into 80 languages.Source: C Dataset Translation · p. 11
  • The full Winoground test set of 800 examples was translated, while other tasks were capped at 500 instances per split.Source: C Dataset Translation · p. 11
  • Table 9 lists the language codes, scripts, and names for the translated datasets.Source: C Dataset Translation · p. 11

D Prompt for Zero-Shot Chinese Vision-Language Tasks

The models were fine-tuned exclusively on English data, and a Chinese dialogue context was added during zero-shot Chinese evaluation to encourage Chinese responses.

D Prompt for Zero-Shot Chinese Vision-Language Tasks

E Template for GPT-4 Evaluation

The evaluation template uses GPT-4 to compare Ying-VLM and baseline responses for quality, with response order swapped across prompts and scores averaged across completions.

E Template for GPT-4 Evaluation

Loading 2306.04387v2…