PaperlayerView exact revision on arXiv

Source-linked AI summary

MANTIS: Interleaved Multi-Image Instruction Tuning

Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, Wenhu Chen

arXiv:2405.01483v3cs.CVcs.AIcs.CL

TL;DR

Multi-image abilities in LMMs remain less developed, while existing systems often depend on massive interleaved web pre-training. Mantis addresses this gap with low-cost instruction tuning on a purpose-built dataset, achieving strong multi-image results while retaining single-image performance.

  • Problem

    Multi-image vision-language capability is less developed than single-image capability, while existing systems rely on computation-intensive pre-training on massive interleaved web data.

  • Method

    Mantis trains a family of models with Mantis-Instruct, a 721K-example interleaved multi-image instruction-tuning dataset covering co-reference, comparison, reasoning, and temporal understanding.

  • Results

    Mantis achieves state-of-the-art performance across five multi-image benchmarks, with Mantis-Idefics2 outperforming Idefics2 by an average of 13 absolute points across seven available benchmarks.

  • Takeaways & Limitations

    The results support low-cost instruction tuning as an effective route to multi-image ability, with strong generalization and preserved single-image performance.

  • Takeaways & Limitations

    Mantis-Instruct contains noise from reformatted source datasets, and its multiple-choice-heavy data contributes to shorter responses and weaker multi-image instruction following.

Abstract

from arXiv · show

Large multimodal models (LMMs) have shown great results in single-image vision language tasks. However, their abilities to solve multi-image visual language tasks is yet to be improved. The existing LMMs like OpenFlamingo, Emu2, and Idefics gain their multi-image ability through pre-training on hundreds of millions of noisy interleaved image-text data from the web, which is neither efficient nor effective. In this paper, we aim to build strong multi-image LMMs via instruction tuning with academic-level resources. Therefore, we meticulously construct Mantis-Instruct containing 721K multi-image instruction data to train a family of Mantis models. The instruction tuning empowers Mantis with different multi-image skills like co-reference, comparison, reasoning, and temporal understanding. We evaluate Mantis on 8 multi-image benchmarks and 6 single-image benchmarks. Mantis-Idefics2 can achieve SoTA results on all the multi-image benchmarks and beat the strongest multi-image baseline, Idefics2-8B by an average of 13 absolute points. Notably, Idefics2-8B was pre-trained on 140M interleaved multi-image data, which is 200x larger than Mantis-Instruct. We observe that Mantis performs equivalently well on the held-in and held-out benchmarks, which shows its generalization ability. We further evaluate Mantis on single-image benchmarks and demonstrate that Mantis also maintains a strong single-image performance on par with CogVLM and Emu2. Our results show that multi-image abilities are not necessarily gained through massive pre-training, instead, they can be gained by low-cost instruction tuning. The training and evaluation of Mantis has paved the road for future work to improve LMMs' multi-image abilities.

1 Introduction

Mantis targets the underdeveloped multi-image abilities of LMMs through instruction tuning rather than massive interleaved pre-training. Its models support four multi-image skills while retaining competitive single-image performance.

  • Multi-image LMM research remains less developed than single-image vision-language research, despite strong single-image capabilities.
  • The targeted skills are co-reference, comparison, reasoning across images, and temporal understanding.These cover reference grounding, cross-image differences and commonalities, inference over multiple pieces of information, and image-sequence dynamics.
  • Existing multi-image LMMs commonly rely on massive interleaved image-text web pre-training, including datasets with 500 million or 140 million examples.The paper frames this computation-heavy approach as difficult to adopt in many open-source training pipelines.
  • Mantis-Instruct contains 721K instances across 14 subsets covering the four multi-image skills, combining existing datasets with four newly curated subsets.The collection includes data for co-reference, reasoning, comparison, and temporal understanding, with image placeholders inserted using heuristics.
  • Mantis combines pretrained language and vision components with a multimodal projector and an explicit interleaved text-image format.The format concatenates image embeddings with text embeddings using image delimiters and serialized image identifiers.
  • Mantis-Eval is manually curated to analyze multi-image abilities, alongside evaluation on multi-image and single-image benchmarks.The evaluation compares Mantis with competitive single-image and multi-image baselines, merging multiple images horizontally for single-image models.
  • Mantis achieves state-of-the-art performance on all evaluated multi-image tasks while preserving competitive single-image performance.Mantis-Idefics2 exceeds Idefics2 by an average of 13 absolute points on seven available benchmarks, and held-in and held-out results are equivalently strong.
  • Low-cost instruction tuning on 721K high-quality examples yields better generalization performance than LMMs intensively pretrained on datasets 100x larger.The models also remain on par with CogVLM and Emu2 on single-image tasks.

2 Mantis: Interleaved Multi-Image Instruction Tuning

Mantis enables interleaved multi-image inputs by adapting LMM architecture, curating a multi-skill instruction dataset, and applying data-organization heuristics. The section describes the architecture, context limits, dataset composition, and curation choices supporting these abilities.

  • 2.1 Method: Mantis modifies LLaVA for multi-image training and inference, while inheriting Idefics2’s structure because Idefics2 natively supports multiple images.
  • 2.1 Method: LLaVA’s architecture uses 576 image tokens per image, allowing at most 14 images for Llama3 with an 8K context window.
  • 2.1 Method: Idefics2 resamples each image to 64 tokens, allowing up to 128 images under an 8K context length.
  • 2.1 Method: Mantis interleaves text and images with explicit begin/end delimiters and serialized image numbers to mark image boundaries and identities.In practice, the delimiters are set to <Image> and </Image>.
  • 2.2 Mantis-Instruct: A large scale multiple image question answering dataset: Mantis-Instruct is assembled from multiple publicly available datasets in a multi-image, multi-turn question-answer format with reported dataset statistics.Table 1 records examples, image counts, conversation turns, and text and text-plus-image token lengths.
  • 2.2 Mantis-Instruct: A large scale multiple image question answering dataset: Co-reference data maps language such as “second image” to the corresponding input image without requiring inference across images.LLaVA-665k-multi and LRV-multi concatenate single-image conversations and add explicit image references.
  • 2.2 Mantis-Instruct: A large scale multiple image question answering dataset: Comparison data covers image quality, visual similarity, and difference description, while reasoning data spans logic, counting, matching, retrieval, and free-form multi-image QA.The reasoning collection combines NLVR2, IconQA, Contrast-Caption, ImageCoDe, and Multi-VQA.
  • 2.2 Mantis-Instruct: A large scale multiple image question answering dataset: Temporal-understanding data includes 14K video-understanding examples plus VIST, NExT-QA, and STAR for image sequences such as videos and comics.

3 Experiments

Mantis is evaluated across held-in and held-out multi-image benchmarks, with comparisons spanning architectures, input formats, pre-training, instruction tuning, and single-image retention. The results show strong multi-image performance and generally preserved single-image ability.

  • Multi-image results: Mantis-Idefics2 nearly matches GPT-4V on the overall multi-image evaluation.
  • Held-in evaluation: 89.71 on NLVR2 and 75.20 on Q-Bench make Mantis-Idefics2 state of the art on both held-in tasks.It also exceeds Idefics2 by 2.84 points on these evaluations.
  • Held-out evaluation: 59.45 on Mantis-Eval, 49.05 on BLINK, and 51.38 on MVBench show strong held-out performance across six unseen benchmarks.The models surpass other baselines on the reported held-out tasks.
  • Pre-training and architecture: Mantis-CLIP and Mantis-SigLIP perform much better than Mantis-Flamingo and similarly to Mantis-Idefics despite lacking massive multi-image pre-training.These results indicate that such pre-training is not necessary for strong multi-image performance.
  • Input format: Sequence-image models outperform horizontal image merging: LLaVA-1.6 reaches only 45.62 on Mantis-Eval and 39.55 on BLINK.Those scores are at least 10 absolute points below Mantis.
  • Single-image evaluation: Mantis-SigLIP improves on MMBench-English, MMMU, and ScienceQA, while Mantis-Idefics2 remains similarly strong to Idefics2 on several single-image tasks.ScienceQA shows some decline for Mantis-Idefics2, with an overall average drop of 4% described as tolerable.

4 Related Works

Related work develops multimodal language models and increasingly addresses multi-image understanding, while highlighting gaps in interleaved training resources and evaluation. Recent datasets and models further demonstrate the value of academic multi-image instruction data.

  • Large Multimodal Models: Large multimodal models combine language with other modalities, with much prior work concentrating on integrating vision knowledge into language models.Examples include efforts involving image, audio, and video, alongside vision-focused models such as BLIP-2.
  • Multi-Image Models: Flamingo targets in-context learning over image/text sequences but lacks free-form interleaved image-text training examples.
  • Multi-Image Models: Multi-image ability has gained attention through models including LLaVA-Next-Interleave and LLaVA-OneVision, which train on large-scale single-image, multi-image, and video understanding data.
  • Multi-Image Models: LLaVA-Next-Interleave and LLaVA-OneVision incorporate Mantis-Instruct as training data, demonstrating the dataset’s value.
  • Benchmarks: Single-image benchmarks are extensive, whereas multi-image evaluation received less attention until recently.Difference description was among the first multi-image tasks to receive large-scale data and established evaluation benchmarks.

5 Conclusion

Mantis is an open multimodal model family trained for interleaved text-image inputs with a dedicated multi-image instruction dataset and evaluation benchmark. It acquires four multi-image skills, reaches state-of-the-art results on reported multi-image benchmarks, and retains decent single-image reasoning performance.

  • The Mantis model family is designed to accept interleaved text-image inputs.
  • Mantis is fine-tuned on 721K interleaved examples covering co-reference, reasoning, comparison, and temporal understanding.
  • Mantis-Eval contains 217 examples averaging 2.5 images per example and is designed to evaluate several multi-image skills.
  • Mantis models achieve state-of-the-art performance on 5 multi-image evaluation benchmarks while preserving decent single-image reasoning performance.

Limitations

The work focuses on multi-image ability, but its dataset can retain source noise and its models tend toward short responses. Improving long-form instruction following and balancing multi-image with single-image ability remain open issues.

  • Mantis-Instruct reformats existing datasets that may retain noise from their original collections.
  • Most Mantis-Instruct examples are multiple-choice questions, contributing to shorter responses and weaker long-form instruction following in multi-image settings.
  • Future work includes adding more long-form multi-image data and better balancing multi-image and single-image abilities.
  • Single-image performance shows a degeneration trend compared with the original model, and conflicts between multi-image and single-image abilities persist.

Societal Impacts

Mantis and Mantis-Instruct could support real-world multimodal analysis and decision-making, including website browsing and travel planning. However, hallucinations, wrong decisions, complex-scenario failures, and misuse remain potential risks.

  • LMMs with multi-image ability could analyze and reason in real-world scenarios to help machines and robots make decisions automatically.
  • Potential applications include assisting with website browsing and travel planning based on multiple maps and pictures.
  • Mantis can still hallucinate, make wrong decisions, and fail to reason across complex real-world scenarios.
  • Open-sourcing the model and dataset creates potential for misuse that licenses and usage terms cannot safely eliminate.

A.1 Details of Mantis-Instruct subsets

Mantis-Instruct combines multiple public datasets and reformats them into multi-image instruction examples covering diverse visual-language skills. Its processing includes synthetic image co-reference, image retrieval, contrastive captioning, difference description, and multi-image reasoning tasks.

  • Dataset format: Mantis-Instruct uses a multi-image, multi-turn QA format assembled from multiple publicly available datasets.Each item contains multiple images and multiple turns of question-answer pairs.
  • Co-reference: LLaVA-665k-multi randomly merges 2 to 4 single-image QA items and adds image denotations to train image co-reference.The synthetic format also supports single-image retention and instruction following.
  • Image retrieval: ImageCoDe requires selecting the correct image from 10 minimally contrastive images using contextual descriptions.The task uses nine similar retrieved images as distractors.
  • Contrastive captioning: Contrast-Caption creates two tasks: matching a caption to an image and generating a caption for a denoted image among 2 to 8 images.Items combine images and captions from ShareGPT-4V and LAION GPT-4V captions.
  • Comparison: Birds-to-Words provides pairs of different-bird images and diverse questions generated from reference answers.The dataset trains descriptions of differences between the images.

A.2 Investigation of Mantis’s capability on open-ended tasks

Mantis is evaluated on open-ended image and video captioning tasks. Training on Mantis-Instruct improves Mantis-Idefics2’s captioning performance, with the authors associating the gain with stronger image differentiation and semantic understanding.

  • Captioning evaluation: Mantis-Idefics2 improves over its previous Idefics2 checkpoint on image and video captioning after training on Mantis-Instruct.The evaluation uses CoCo-2017-Lite for image captioning and Vatex for video captioning.
  • Tasks: Mantis’s open-ended evaluation reports results for image captioning on CoCo-2017-Lite and video captioning on Vatex.The results are presented in Tables 9 and 10.
  • Interpretation: The authors attribute the improvement to dataset designs that encourage differentiating images and understanding semantic visual information.They connect this design to contrastive learning that is limited in many vision-language pre-training datasets.

A.3 Investigation of Mantis’s capability on long-context scenarios

Mantis maintains competitive performance in long-context multimodal settings and remains stable as the number of sampled video frames changes. On MileBench, it approaches GPT-4V despite examples averaging 15 images and reaching 9,000 tokens.

  • Frame scaling: Mantis-Idefics2’s MVBench performance is stable across 2, 4, 8, and 16 uniformly sampled frames.Accuracy rises from 48.38 with 2 frames to 51.38 with 8 frames; the 16-frame result is one point lower than the preceding result.
  • Frame scaling: Using only 2 frames yields 48.38 accuracy on MVBench, while 8 frames yield 51.38 accuracy.The authors note that many questions can be answered from keyframes.
  • MileBench: 47.5 on MileBench leaves Mantis-SIGLIP 5.5 points behind GPT-4V in examples averaging 15 images and spanning 1,000 to 9,000 tokens.The authors view this as evidence of competitive multimodal long-context ability.

A.4 Comparison with MIMIC-IT

A controlled comparison shows that Mantis-Flamingo outperforms Otter when both share the same pre-training data but use different instruction-tuning datasets. The authors attribute this advantage to Mantis-Instruct’s coverage of four essential multi-image abilities.

  • Controlled comparison: Mantis-Flamingo performs better than Otter on all reported multi-image benchmarks under the same pre-trained model.The comparison isolates instruction-tuning data: Mantis-Instruct for Mantis-Flamingo versus MIMIC-IT for Otter.
  • Dataset comparison: The authors attribute Mantis-Instruct’s advantage to its design around four essential multi-image reasoning abilities.They contrast this with MIMIC-IT’s primary focus on multimodal in-context reasoning.
Loading 2405.01483v3…