Source-linked AI summary
MIMIC-IT: Multi-Modal In-Context Instruction Tuning
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, Ziwei Liu
TL;DR
Interactive vision-language assistants lack sufficient diverse, creative, and multimodal instruction-response data for complex visual tasks. MIMIC-IT addresses this gap with a large dataset and automated Syphus pipeline, and Otter shows strong benchmark, human-evaluation, and in-context-learning performance. The authors also note that language hallucinations in ChatGPT can produce incorrect generated responses.
Problem
Existing vision-language instruction-response resources are limited in quantity, diversity, creativity, visual inputs, and multimodal in-context information.
Method
MIMIC-IT provides multimodal in-context instruction-response data from diverse images and videos, generated through Syphus and used to train Otter on OpenFlamingo.
Results
Otter achieves state-of-the-art perception and reasoning performance, strong human-evaluation results, and superior few-shot in-context learning over OpenFlamingo.
Takeaways & Limitations
MIMIC-IT supports multimodal perception, reasoning, planning, interactive instruction following, and in-context learning across diverse visual scenes.
Takeaways & Limitations
ChatGPT may generate incorrect instruction-response pairs because it is prone to language hallucinations.
Abstract
from arXiv · showhide
High-quality instructions and responses are essential for the zero-shot performance of large language models on interactive natural language tasks. For interactive vision-language tasks involving intricate visual scenes, a large quantity of diverse and creative instruction-response pairs should be imperative to tune vision-language models (VLMs). Nevertheless, the current availability of vision-language instruction-response pairs in terms of quantity, diversity, and creativity remains limited, posing challenges to the generalization of interactive VLMs. Here we present MultI-Modal In-Context Instruction Tuning (MIMIC-IT), a dataset comprising 2.8 million multimodal instruction-response pairs, with 2.2 million unique instructions derived from images and videos. Each pair is accompanied by multi-modal in-context information, forming conversational contexts aimed at empowering VLMs in perception, reasoning, and planning. The instruction-response collection process, dubbed as Syphus, is scaled using an automatic annotation pipeline that combines human expertise with GPT's capabilities. Using the MIMIC-IT dataset, we train a large VLM named Otter. Based on extensive evaluations conducted on vision-language benchmarks, it has been observed that Otter demonstrates remarkable proficiency in multi-modal perception, reasoning, and in-context learning. Human evaluation reveals it effectively aligns with the user's intentions. We release the MIMIC-IT dataset, instruction-response collection pipeline, benchmarks, and the Otter model.
1 Introduction
MIMIC-IT addresses limitations in existing multimodal instruction data by providing diverse visual inputs and multimodal conversational context, then uses them to train Otter for interactive vision-language tasks.
- Existing multimodal instruction data requires greater diversity and quality to support assistants that handle multiple modalities.
- LLaVA-Instruct-150K is limited by COCO-only visual diversity, single-image inputs, and language-only in-context information.
- MIMIC-IT supports diverse images and videos, multiple visual inputs, and multimodal in-context examples for perception, reasoning, and planning.Its Syphus pipeline uses system messages, visual annotations, and in-context examples to generate instruction-response pairs from visual context.
- Otter, trained on MIMIC-IT using OpenFlamingo, achieves strongest performance in ChatGPT evaluation, highest Elo in human evaluation, and superior few-shot results over OpenFlamingo.
- MIMIC-IT contains ~2.8M multimodal in-context instruction-response pairs with 2.2 million unique instructions across real-life scenes.
2 Related Work
Related work spans multimodal instruction datasets, multimodal aggregation systems, and end-to-end trainable foundation models. Otter extends OpenFlamingo with multimodal in-context instruction tuning.
- Earlier multimodal instruction datasets covered visual understanding and reasoning, while Mini-GPT4 combined several image-text datasets with handwritten instruction templates.
- LLaVA-Instruct-150K improved instruction-data quality by using self-instruct and GPT-4-based generation.
- Multimodal systems include aggregation approaches that connect expert models and end-to-end trainable models that integrate modalities.Aggregation approaches use language to call task-specific experts but cannot train each model individually on new tasks.
- Otter is presented as the first open-sourced model based on OpenFlamingo to demonstrate multimodal in-context instruction tuning.
3 Multi-modal In-context Instruction Tuning Dataset
MIMIC-IT is designed as a large multimodal instruction-tuning dataset with multiple visual inputs, multimodal in-context examples, automated multilingual generation, and diverse scene-oriented tasks. Its statistics emphasize scale and variation in instructions, responses, visual inputs, and contextual examples.
- 3 Multi-modal In-context Instruction Tuning Dataset: MIMIC-IT extends single-image, language-only context formats by supporting multiple images or videos and multimodal in-context information.Its in-context information can include both visual inputs and language inputs.
- 3.1 MIMIC-IT Data Format: MIMIC-IT represents each query with an instruction-response pair and a set of N images or videos, together with corresponding in-context examples.Videos are treated as ordered sequences of images.
- 3.1 MIMIC-IT Data Format: The model formulation conditions response generation on the query instruction, visual input, and task-dependent organized in-context examples.The dataset represents a query example alongside its corresponding contextual examples.
- 3.2 Sythus: Automatic Instruction-Response Generation Pipeline: Sythus is an automated pipeline that uses system messages, visual annotations, and in-context examples to generate high-quality instruction-response pairs in eight languages.A cold-start stage identifies a system message and in-context example before the generation steps.
- 3.3 Visual Data Exploration: The dataset combines seven image and video datasets spanning general scenes, egocentric views, indoor RGB-D data, visual differences, stories, dense captions, and TV narratives.The general-scene tasks include LLaVA-Interleaved, Spot The Difference, Visual Story Telling, and Dense Captions.
- 3.3.1 General Scene Understanding: LLaVA-Interleaved adds ten retrieved in-context examples to each LLaVA-Instruct-150K instruction-response pair using text-text or image-image similarity.This construction specifically targets learning with in-context examples.
- 3.4 Dataset Statistics: The dataset contains over 2.8 million instruction-response pairs and 2.2 million unique instructions, with every pair including multimodal and language-only in-context examples.The dataset also varies instruction and response lengths, images per instruction, and in-context examples per instruction.
- 3.4 Dataset Statistics: Response sentence structures are more diverse than instruction structures, based on analyses of root verbs and direct noun objects.The analysis plots the top 20 root verbs and their top four direct noun objects.
4 Empricial Evaluation
The evaluation demonstrates Otter’s capabilities in scene understanding, reasoning, multi-round interaction, egocentric assistance, benchmark performance, human alignment, and in-context learning.
- ChatGPT Evaluation: ChatGPT-based MMAGIBench evaluation finds Otter achieves the highest average accuracy across perception and reasoning tasks.
- Usage Examples and Demonstrations: Otter supports scene comprehension, reasoning, and multi-round conversation after training on MIMIC-IT.
- Egocentric Visual Assistant: Otter-E uses egocentric videos and sequential images to support first-person scene perception, task planning, and AR-headset assistance.
- ChatGPT Evaluation: MMAGIBench evaluates VLM perception across scene, object, OCR, celebrity, and location recognition, and reasoning across attributes, relations, and future prediction.
- Human Evaluation: Human evaluation shows Otter has superior usefulness and alignment, achieving the highest Elo rating among recent VLMs.
- Few-shot In-context Learning Metric Evaluation: Otter outperforms OpenFlamingo by a substantial margin on COCO Caption CIDEr few-shot evaluation, with a marginal zero-shot gain after finetuning.
5 Discussion
The discussion identifies hallucination risk in ChatGPT-generated instruction data and outlines future expansion toward embodied AI and more trustworthy generation.
- ChatGPT may generate incorrect responses because it is prone to language hallucinations during self-instruct data generation.
- Future work includes adding embodied AI datasets and improving instruction collection with more trustworthy language models or generation techniques.
- MIMIC-IT covers diverse visual scenes and creative instructions in eight languages, while Otter achieves strong perception, reasoning, and human-evaluation performance.
A Total Cost and ChatGPT Version
The appendix reports the ChatGPT version, token usage, and estimated cost used to construct MIMIC-IT.
- MIMIC-IT construction used ChatGPT-0301 and queried 1,006,746,240 tokens at an estimated total cost of $20134.9248.3.
B Content Copyright and License
The appendix lists the licenses associated with the visual datasets and instruction-response data used in the work.
- The license table lists seven visual datasets and their corresponding image and instruction-response license statuses.
C Sythus: Automatic Instruction Generation Pipeline
The pipeline applies GPT-based safety filtering and expands English instruction-response pairs into seven additional languages.
- GPT-based filtering removes outputs involving unsafe, unfair, stereotypical, misleading, or unreliable content.
- English instruction-response pairs are translated into Chinese, Japanese, Spanish, German, French, Korean, and Arabic.
D Annotation Prompt
The annotation prompts combine system messages, in-context examples, and dataset-specific exemplars to guide ChatGPT queries across multimodal tasks.
- Prompts contain a system message and an in-context example for querying ChatGPT.
- A multilingual translation query is illustrated with a tea-related instruction and response example.
- Dataset-specific prompt exemplars are provided for TV captions, dense captioning, Ego4D, indoor event planning, spot-the-difference, and visual storytelling.