Source-linked AI summary
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, Chunyuan Li
TL;DR
Open-source LMMs have focused mainly on single-image tasks, leaving multi-image capabilities and cross-scenario generalization less explored. LLaVA-NeXT-Interleave unifies Multi-image, Multi-frame, Multi-view, and Multi-patch settings through interleaved data, supported by M4-Instruct and LLaVA-Interleave Bench. It achieves leading results across multi-image, video, and 3D benchmarks while maintaining single-image performance and showing cross-setting task transfer.
Problem
Open-source LMMs primarily focus on single-image tasks, while multi-image scenarios and unified modeling across visual settings remain less explored.
Method
The model uses image-text interleaving as a universal data template and trains on M4-Instruct across multi-image, video, 3D, and single-image settings.
Results
LLaVA-NeXT-Interleave achieves leading results across multi-image, video, and 3D benchmarks while maintaining single-image performance.
Takeaways & Limitations
Joint cross-scenario training exhibits task transfer across settings and modalities, including transferring image difference spotting to videos.
Abstract
from arXiv · showhide
Visual instruction tuning has made considerable strides in enhancing the capabilities of Large Multimodal Models (LMMs). However, existing open LMMs largely focus on single-image tasks, their applications to multi-image scenarios remains less explored. Additionally, prior LMM research separately tackles different scenarios, leaving it impossible to generalize cross scenarios with new emerging capabilities. To this end, we introduce LLaVA-NeXT-Interleave, which simultaneously tackles Multi-image, Multi-frame (video), Multi-view (3D), and Multi-patch (single-image) scenarios in LMMs. To enable these capabilities, we regard the interleaved data format as a general template and compile the M4-Instruct dataset with 1,177.6k samples, spanning 4 primary domains with 14 tasks and 41 datasets. We also curate the LLaVA-Interleave Bench to comprehensively evaluate the multi-image performance of LMMs. Through extensive experiments, LLaVA-NeXT-Interleave achieves leading results in multi-image, video, and 3D benchmarks, while maintaining the performance of single-image tasks. Besides, our model also exhibits several emerging capabilities, e.g., transferring tasks across different settings and modalities. Code is available at https://github.com/LLaVA-VL/LLaVA-NeXT
1. Introduction
LLaVA-NeXT-Interleave addresses the limited exploration of multi-image LMMs and fragmented handling of video, 3D, and single-image settings by unifying them through interleaved data. It introduces M4-Instruct and LLaVA-Interleave Bench, achieving leading multi-image results while maintaining single-image performance and enabling cross-task transfer.
- Research gap: Open-source LMMs have primarily optimized single-image tasks, while multi-image applications and unified cross-scenario modeling remain underexplored.Separate models for multi-image, video, and 3D tasks are labor-intensive, time-consuming, and difficult to scale.
- Unified approach: The image-text interleaved format represents single-image, multi-image, video, and 3D inputs as unified data settings.The paper names these four settings M4: Multi-image, Multi-frame, Multi-view, and Multi-patch.
- Data and evaluation: M4-Instruct contains 1177.6 samples spanning 4 domains, 14 tasks, and 41 datasets, while LLaVA-Interleave Bench combines 7 newly collected and 13 existing benchmarks.The dataset supports training and the benchmark evaluates multi-image performance across in- and out-domain settings.
- Results: A single LLaVA-NeXT-Interleave model achieves leading results across different multi-image tasks compared with the previous state of the art while maintaining single-image performance.The paper presents this as a headline outcome of the unified training approach.
- Emerging capabilities: Joint training across diverse tasks yields emerging cross-task transfer, including transferring image difference spotting to videos.This capability is reported across different settings and modalities.
2. Related Work
Prior interleaved LMM work covers multimodal in-context learning and selected multi-image applications, while benchmarks often target individual domains. LLaVA-NeXT-Interleave broadens instruction tuning across video, 3D, and single-image settings and introduces a broader high-quality evaluation suite.
- Interleaved LMMs: Interleaved image-text data supports multimodal in-context learning and instruction following for real-world multi-image applications.In-context learning interleaves image-text examples as task demonstrations for few-shot adaptation at inference.
- Interleaved LMMs: Open-source interleaved LMMs have developed multi-image proficiency, but instruction tuning for diverse real-world multi-image applications remains less explored.The related work contrasts instruction tuning with multimodal pre-training and identifies Mantis as an exception.
- Data format: M4-Instruct organizes visual storytelling, video, 3D, and single-image data in an interleaved format for unified processing.Figure 2 presents multi-image storytelling alongside multi-frame, multi-view, and single-image examples.
- Benchmarks: Earlier benchmarks commonly focus on specific domains, whereas LLaVA-Interleave Bench evaluates a collection of high-quality multi-image scenarios.The proposed benchmark includes specific and general categories and addresses quality concerns associated with prior broad collections.
3. Interleaved Multi-image Tasks & Data
The paper represents multi-image, video, 3D, and single-image inputs through a unified interleaved format, then builds M4-Instruct and LLaVA-Interleave Bench to train and evaluate these capabilities.
- Task Overview: Interleaved multi-image formatting unifies multi-image, multi-frame, multi-view, and multi-patch tasks within one data input representation.Video is represented as sampled frames, 3D as multi-view images, and high-resolution single images as multiple patches.
- Task Overview: Multi-image training covers 12 challenging tasks, including difference spotting, visual storytelling, image-editing instruction generation, dialogue, puzzles, and low-level assessment.
- Task Overview: Multi-frame training focuses on video detailed captioning and video VQA while preserving temporal visual cues through sampled frame sequences.
- Task Overview: 3D training includes embodied VQA and 3D scene VQA, using visual correspondence and disparity across viewpoints to represent spatial information.
- M4-Instruct: 1,177.6K M4-Instruct instances span four domains, 14 tasks, and 41 datasets, including multi-patch data intended to preserve single-image performance.
- LLaVA-Interleave Bench: The LLaVA-Interleave Bench contains 13 challenging tasks with 17K instances, divided into 12.9K in-domain and 4.1K out-of-domain evaluation examples.Out-domain evaluation targets generalization through mathematical and scientific comprehension tasks plus existing benchmarks.
4. Interleaved Visual Instruction Tuning
LLaVA-NeXT-Interleave extends a single-image model through interleaved visual instruction tuning across mixed formats and four data scenarios. The training design aims to improve multi-image performance while supporting flexible input formats and complementary task learning.
- Architecture: The model follows a general LLaVA-NeXT architecture with a vision encoder, intermediate projector, and large language model.
- Technique 1: Continue training from single-image models: Training continues from LLaVA-NeXT-Image to leverage pretrained single-image captioning and fine-tuning proficiency for more challenging interleaved tasks.
- Technique 2: Mixed Interleaved data formats during training: The training mixes an in-the-front format, placing image tokens before the prompt, with an interleaved format that preserves their original positions.
- Technique 2: Mixed Interleaved data formats during training: Using both image-token formats supports more flexible inference modes and robustness to different input formats.
- Technique 3: Combining different data scenarios improves individual task performance: M4-Instruct jointly tunes multi-image, multi-frame, multi-view, and multi-patch data so distinct scenarios can provide complementary semantics.
5. Experiments
The experiments evaluate LLaVA-NeXT-Interleave across multi-image, video, 3D, and single-image settings, with ablations of its training techniques. The model achieves leading results across multiple scenarios while preserving single-image performance.
- Evaluation Schemes: The evaluation covers multi-image, multi-frame video, multi-view 3D, and multi-patch single-image scenarios.Multi-image evaluation uses the LLaVA-Interleave Bench; video and 3D use established and curated benchmarks.
- Multi-image Results: LLaVA-NeXT-Interleave surpasses previous open-source models on average multi-image performance across both in-domain and out-domain benchmarks.It is also comparable to GPT-4V on Mantis-Eval and BLINK.
- Multi-frame (Video) Results: After DPO training, the 7B model attains state-of-the-art performance on VDD and VideoChatGPT, surpassing the previous 34B LLaVA-NeXT-Video.The results demonstrate temporal understanding and reasoning across sequential frames.
- Multi-view (3D) Results: The model obtains leading results on five indoor and outdoor 3D benchmarks using only multi-view images, while maintaining single-image performance.Single-image data is incorporated through 307k samples, representing 40% of the original LLaVA-NeXT single-image data.
- Ablations of Proposed Techniques: Initialization from a strong single-image checkpoint consistently improves interleaved multi-image performance, while mixed-format and combined-scenario training improve results across input settings.Progressively adding single-image and multi-image data to video training also enhances performance compared with individual visual scenarios.
6. Emerging Capabilities
The model demonstrates capabilities that were not explicitly trained, including transferring tasks between single-image, multi-image, and video settings. It also handles several real-world applications absent from the interleaved training data.
- Task Transfer from Single-image to Multi-image: The model transfers single-image reasoning about humorous content to multiple images, although that task was excluded from multi-image training.The authors attribute this capability probably to composing single-image capability with multi-image VQA training.
- Task Transfer from Image to Video: The model performs the Twitter-post task on video despite training containing the task only for multi-image inputs.The authors describe this capability as probably composed from multi-image Twitter-post and video VQA training.
- Real-world Applications: The model demonstrates painting-style recognition, PPT summarization and question answering, and multi-document VQA without explicit training examples for these scenarios.These examples are presented as real-world applications showing generalization potential.
7. Conclusion
The paper concludes that interleaved data can unify multi-image, video, 3D, and single-image scenarios in one LMM. It reports strong performance across domains and emerging cross-task capabilities.
- Conclusion: LLaVA-NeXT-Interleave integrates multi-image, video, 3D, and single-image scenarios through an interleaved data format.The paper presents this as a cohesive approach to varied visual tasks.
- Conclusion: M4-Instruct and LLaVA-Interleave Bench provide resources for training and evaluating LMMs across multiple visual domains.The conclusion characterizes these resources as a foundation for multi-domain training and evaluation.
- Conclusion: The model sets state-of-the-art results on multi-image tasks while maintaining exceptional single-image performance.The conclusion also reports promising emerging capabilities such as cross-task transfer.
- Conclusion: The reported cross-task transfer suggests versatility and potential for broader visual applications.This conclusion is tied to the model's emerging capabilities across settings.
A. Data Statistics
The paper points readers to tables containing detailed statistics for M4-Instruct and LLaVA-Interleave Bench.
- Data Statistics: Detailed data statistics for M4-Instruct and LLaVA-Interleave Bench are provided in Tables 15 and 16, respectively.
B.1. Pool vs not Pool Vision Tokens for video tasks.
The study compares video training and inference with image-token pooling to 1/4 against no pooling. Pooling reduces visual-token count while testing its effect on video tasks.
- Pooling strategy: Pooling feature-map width and height to 1/2 reduces the total image-token count to 1/4.The strategy is adopted similarly to LLaVA-NEXT-Video.
- Experimental settings: The experiment trains and infers the model under both pooling-to-1/4 and no-pooling settings.Both settings use ShareGPTVideo-Caption+QA(255K) data.
- Experimental goal: The ablation studies how image-token pooling affects video-task performance while trading off token count.
B.2. Impact of video DPO training on other tasks.
The paper evaluates whether video DPO training transfers to other tasks. Video DPO substantially improves video performance but slightly affects performance on other tasks.
- Impact on video: Video DPO significantly improves video performance, as shown in Table 2.
- Impact on other tasks: Video DPO slightly impacts the performance of other tasks.The paper reports this cross-task effect in Table 14.
- Evaluation: Table 14 reports the cross-task ablation using metrics including QB, Act, MVB, VDD, MME*, and SQA.MME* is converted to accuracy by summing perception and cognition scores and dividing by 2800.