Source-linked AI summary
Otter: A Multi-Modal Model with In-Context Instruction Tuning
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, Ziwei Liu
TL;DR
Existing multimodal instruction tuning provides limited use of combined visual and textual in-context examples. Otter addresses this gap with Flamingo-based multimodal in-context instruction tuning and the MIMIC-IT dataset, with evaluations reporting improved training efficiency, generalization, and multimodal-task performance. The paper also identifies fine-grained OCR as a limitation because of the pretrained vision encoder’s 224×224 resolution.
Problem
Existing multimodal instruction tuning focuses on individual instructions or previous dialogues, with little discussion of using both images and text as in-context examples.
Method
Otter uses a Flamingo-based architecture with multimodal in-context instruction tuning, supported by MIMIC-IT’s diverse image- and video-based instruction-response data.
Results
Evaluations report improved training efficiency, generalization, robustness, and performance across multimodal benchmarks and complex video and multi-image tasks.
Takeaways & Limitations
MIMIC-IT’s diverse multimodal contexts support Otter’s potential as a general-purpose assistant for complex video and multi-image understanding.
Takeaways & Limitations
Fine-grained OCR remains limited by the pretrained vision encoder’s relatively low 224×224 resolution.
Abstract
from arXiv · showhide
Recent advances in Large Multimodal Models (LMMs) have unveiled great potential as visual assistants. However, most existing works focus on responding to individual instructions or using previous dialogues for contextual understanding. There is little discussion on employing both images and text as in-context examples to enhance the instruction following capability. To bridge this gap, we introduce the \textbf{Otter} model to leverage both textual and visual in-context examples for instruction tuning. Specifically, Otter builds upon Flamingo with Perceiver architecture, and has been instruction tuned for general purpose multi-modal assistant. Otter seamlessly processes multi-modal inputs, supporting modalities including text, multiple images, and dynamic video content. To support the training of Otter, we present the \textbf{MIMIC-IT} (\textbf{M}ult\textbf{I}-\textbf{M}odal \textbf{I}n-\textbf{C}ontext \textbf{I}nstruction \textbf{T}uning) dataset, which encompasses over 3 million multi-modal instruction-response pairs, including approximately 2.2 million unique instructions across a broad spectrum of images and videos. MIMIC-IT has been carefully curated to feature a diverse array of in-context examples for each entry. Comprehensive evaluations suggest that instruction tuning with these in-context examples substantially enhances model convergence and generalization capabilities. Notably, the extensive scenario coverage provided by the MIMIC-IT dataset empowers the Otter model to excel in tasks involving complex video and multi-image understanding.
1 INTRODUCTION
Otter addresses the limited use of combined visual and textual in-context examples in multimodal instruction tuning. Built on Flamingo and supported by the MIMIC-IT dataset, it targets stronger instruction following, convergence, generalization, and complex video and multi-image understanding.
- Existing multimodal instruction tuning often uses single-image instructions or preceding dialogues, overlooking combined image-text contextual examples.
- Otter inherits Flamingo’s structure and uses textual and visual in-context examples to support general-purpose multimodal assistance.The approach aligns instruction tuning with Flamingo’s image-text interleaved pretraining paradigm.
- Over 3 million multimodal instruction-response pairs, including approximately 2.2 million unique instructions, comprise the MIMIC-IT dataset.The dataset spans images and videos and provides diverse multimodal contextual information for each instance.
- Diverse scenario coverage enables Otter to address complex video and multi-image comprehension tasks.The dataset’s contextual variety is described as supporting the model’s adaptability across these domains.
- Comprehensive evaluations report benefits from multimodal in-context instruction tuning in training efficiency, generalization, robustness, and performance.The reported gains extend across image and video benchmarks and diverse video and multi-image tasks.
2 RELATED WORK
Related work connects instruction tuning and in-context learning in language and multimodal models, while identifying limited exploration of their integration for multimodal instruction tuning.
- In-context learning adapts tasks through contextual input without retraining, but its underlying mechanics remain underexplored.
- Studies suggest connections between in-context learning and instruction tuning, including the use of noisy pretrained-language-model instances for instruction tuning.
- Multimodal instruction-tuning research enables models to process and respond to multimodal instructions across visual comprehension and reasoning tasks.
- The integration of in-context learning with multimodal instruction tuning remains largely unexplored, aside from Flamingo’s few-shot capabilities after image-text interleaved pretraining.
3 OTTER MODEL
Otter is built on OpenFlamingo with a vision encoder, language decoder, Perceiver resampler, and cross-gated attention. Its training mixes visual tokens with text for next-token prediction, while its image and video variants support multiple images and temporal frames.
- 3 OTTER MODEL: Otter is based on OpenFlamingo and integrates CLIP vision models with language models through Flamingo-style visual processing components.
- 3 OTTER MODEL: The Perceiver resampler converts image or video features into a fixed set of visual tokens before they are integrated with language processing.
- 3 OTTER MODEL: Only the Perceiver resampler and cross-gated attention modules are trained within pretrained OpenFlamingo-9B, totaling 1.4B parameters.
- 3 OTTER MODEL: Training mixes visual tokens with text and conditions later language-model layers through cross-gated attention under a next-token prediction objective.
- 3 OTTER MODEL: Otter-Image uses the N dimension for contextual images, while Otter-Video uses the T dimension for sequential frames and adds time embeddings.
4 MIMIC-IT DATASET
MIMIC-IT structures multimodal instruction tuning around queries paired with visual inputs and diverse in-context examples. It combines broad task coverage, videos, and dataset analyses with documented privacy, bias, and safety considerations.
- Dataset format: MIMIC-IT represents each query with an instruction, response, associated images or videos, and multiple sets of in-context examples.Videos are treated as sequential images, and the context function organizes examples according to the current query.
- Dataset format: The dataset integrates query examples with corresponding in-context examples through a task-specific context function.Different organizational strategies are required depending on the query.
- Dataset scope: MIMIC-IT covers eight diverse subsets, including instruction following with examples, image-sequence analysis, image comparison, television narratives, and egocentric video assistance.The subsets target varied multimodal scenarios and daily user-assistant interactions.
- Dataset scope: MIMIC-IT contains 3M instruction-tuning examples, video data, in-context instruction tuning, multilingual capabilities, and data from seven diverse contexts.The comparison describes these as distinguishing characteristics relative to other multimodal instruction datasets.
- Dataset statistics: Instruction and response analyses examine root verb-noun structures, length, images per instruction, and in-context examples per instruction, finding more diverse response sentence structures.The analysis retains 25% of Ego4D instructions for a more balanced distribution.
- Safety and ethics: Dataset curation raises privacy, bias, and safety concerns involving public-domain sources, consent, demographic distributions, and generated instruction-response content.The authors report race and gender classification on a uniform 5% sample and adherence to Microsoft Azure’s ChatGPT4 content policy.
5 CHATGPT-ASSISTED DATASET GENERATION
The paper uses the Syphus pipeline and ChatGPT to generate multilingual, scenario-specific multimodal instruction-response data with in-context examples. Human checking, automated correction, and invalid-sample tracking address recurring generation errors.
- Pipeline: Syphus generates multilingual instruction-response pairs from visual content and original annotations using ChatGPT.The pipeline builds on LLaVA and generates data in multiple languages.
- Pipeline: System messages, visual annotations, and in-context examples guide ChatGPT’s tone, style, content, and contextual responses.Visual annotations include bounding boxes and image descriptions.
- Scenario Coverage: Figures 5–8 illustrate diverse scenarios across dataset subsets, emphasizing the breadth and contextual variety of the generated data.The examples cover distinct characteristics of multiple subsets and dialogue settings.
- Dataset Cleaning: Most generated pairs follow the specified format, content, style, and pronoun usage, but some deviate from intended scenario conventions.The E4D example requires second-person questions and first-person AR-assistant responses.
- Dataset Cleaning: Human cross-checking, regular-expression fixes, invalid.json logging, and ChatGPT regeneration are used to handle fixable and non-batch-correctable errors.Two human reviewers each spent 15 hours checking error patterns across datasets.
6 EXPERIMENTS
Experiments evaluate dataset composition, task descriptions, in-context examples, convergence, and applications across image and video settings. Results indicate that contextual information improves convergence and preserves few-shot gains, while task-description benefits depend on the benchmark.
- Benchmark Evaluation: Otter-Image and Otter-Video are evaluated across image and video benchmarks, including POPE, MM-Vet, MMBench, MathVista, MME, MSVD-QA, ActivityNet-QA, and MSRVTT-Caption.Video evaluation includes zero-shot ChatGPT-assisted scoring and few-shot captioning.
- Task Descriptions: Task descriptions produce lower training loss and improved convergence after 20K steps compared with training without task descriptions.The divergence becomes evident after initially similar loss curves.
- Task Descriptions: 1212.9 →1414.5 on MME shows a gain from controlling answer format through task descriptions, whereas the same effect is not observed on MM-Vet.MME-Cog uses yes/no answers, while MM-Vet uses freeform responses.
- In-Context Examples: Otter without ICIT excels at 0-shot evaluation but declines as examples increase, while Otter with ICIT improves with more shots.This contrasts with base Flamingo and OpenFlamingo models, which often benefit from additional in-context examples.
- Diverse Applications: MIMIC-IT supports applications including video captioning, video assistance, egocentric scene interpretation, indoor navigation, and planning.Figure 13 demonstrates responses across video and egocentric scenarios.
7 CONCLUSION
Otter is presented as a multimodal model with in-context instruction tuning, supported by the MIMIC-IT dataset. Empirical results indicate improved training efficiency and multimodal performance, while fine-grained OCR remains limited by the pretrained vision encoder’s resolution.
- Otter is a multimodal model with in-context instruction tuning, driven by MIMIC-IT’s multimodal contextual information.
- The evaluation includes both fully correct and incorrect video-response matches, receiving scores of 100 and 0 respectively.The athlete-gender response matches the ground truth, whereas the billiards response does not answer what happened afterward.
- Empirical results demonstrate superior training efficiency and performance in multimodal tasks.
- Fine-grained OCR remains limited because the pretrained vision encoder uses relatively low resolution of 224×224.