Source-linked AI summary
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, Jingren Zhou
TL;DR
Long image sequences expose efficiency, context, and visual-detail limitations in existing multimodal models. mPLUG-Owl3 uses Hyper Attention to integrate language and visual sequences efficiently, achieving broad benchmark strength and strong ultra-long-sequence performance, while video examples still reveal hallucinations.
Problem
Existing methods struggle to model long image sequences efficiently while preserving fine-grained visual information.
Method
mPLUG-Owl3 combines a visual encoder, projection layer, decoder-only language model, and Hyper Attention blocks for language-guided visual sequence integration.
Results
mPLUG-Owl3 achieves state-of-the-art results on 14 of 20 same-size benchmarks and performs strongly on ultra-long visual sequences.
Takeaways & Limitations
The architecture provides an efficient approach for single-image, multi-image, video, and ultra-long visual-sequence understanding.
Takeaways & Limitations
Long-video examples still contain hallucinations and temporal misinterpretations, such as confusing reefs with a beach and daytime boat activity with nighttime.
Abstract
from arXiv · showhide
Multi-modal Large Language Models (MLLMs) have demonstrated remarkable capabilities in executing instructions for a variety of single-image tasks. Despite this progress, significant challenges remain in modeling long image sequences. In this work, we introduce the versatile multi-modal large language model, mPLUG-Owl3, which enhances the capability for long image-sequence understanding in scenarios that incorporate retrieved image-text knowledge, interleaved image-text, and lengthy videos. Specifically, we propose novel hyper attention blocks to efficiently integrate vision and language into a common language-guided semantic space, thereby facilitating the processing of extended multi-image scenarios. Extensive experimental results suggest that mPLUG-Owl3 achieves state-of-the-art performance among models with a similar size on single-image, multi-image, and video benchmarks. Moreover, we propose a challenging long visual sequence evaluation named Distractor Resistance to assess the ability of models to maintain focus amidst distractions. Finally, with the proposed architecture, mPLUG-Owl3 demonstrates outstanding performance on ultra-long visual sequence inputs. We hope that mPLUG-Owl3 can contribute to the development of more efficient and powerful multimodal large language models.
1 Introduction
MLLMs are strong at single-image tasks but remain challenged by long image sequences, where existing approaches trade computational efficiency against visual detail. mPLUG-Owl3 addresses this with Hyper Attention and reports broad benchmark performance plus a dedicated distractor-resistance evaluation.
- Existing MLLMs primarily align language models with visual encoders using large-scale image-text data, yielding strong single-image understanding.
- Long image-sequence understanding is needed for multi-image reasoning, multimodal retrieval, video, agents, and multi-document question answering.
- Directly inserting visual features into text sequences substantially increases inference latency and memory usage.
- mPLUG-Owl3 introduces Hyper Attention blocks that parallel cross-attention with self-attention and reuse language queries to select complementary visual features.
- 14 out of 20 benchmarks show state-of-the-art results for mPLUG-Owl3 among models of the same size.The evaluation covers single-image, multi-image, and video tasks, and introduces Distractor Resistance for long visual sequences.
- mPLUG-Owl3 demonstrates outstanding performance and high execution efficiency on ultra-long visual sequence inputs.
2 mPLUG-Owl3
mPLUG-Owl3 combines a visual encoder, projection layer, and decoder-only language model with lightweight Hyper Attention blocks. The design uses sparse cross-attention, language-guided visual selection, and multimodal positional modeling for efficient interleaved vision-language processing.
- The architecture comprises a visual encoder, linear projection layer, and decoder-only language model, with Hyper Attention enabling adaptive visual sequence utilization.Siglip-400m and Qwen2 are used unless otherwise specified.
- Unlike direct visual-token insertion, cross-attention feeds visual information into the language model while avoiding context-window, memory, and computational overhead.
- Hyper Attention sparsely extends selected transformer blocks with cross-attention parallel to self-attention, reducing additional computation.
- MI-Rope represents image positions in interleaved sequences, addressing the lack of visual positional embeddings in prior cross-attention models.
- The block reuses self-attention queries for visual cross-attention, preserving specific visual information and supplementing content according to textual semantics.
- An adaptive gate derives information-transfer control from textual features rather than using only a learnable scale.
3 Implement Details
mPLUG-Owl3 is trained in three stages that progress from image-text alignment to multi-image and video understanding, then broad supervised instruction tuning. The implementation combines diverse data sources with image cropping, frame sampling, and distributed training.
- Training Paradigm: Training proceeds through image-text pre-training, multi-image and video training, and supervised fine-tuning on single- and multi-image tasks.
- Training Paradigm: Pre-training uses a randomly sampled subset of 41 million image-text pairs from public datasets.
- Training Paradigm: Multi-image training includes interleaved, text-rich, and video data to teach image selection, text reconstruction, and high-resolution image structure.
- Training Paradigm: Video training uses ShareGPTVideo and VATEX data, with 8 frames sampled per video.
- Training Paradigm: Tensor parallelism splits the model into four parts and reduces single-GPU memory usage to 32–40 GB.
- Training Paradigm: Supervised fine-tuning combines diverse image, multi-image, video question-answering, and captioning datasets.
- Input Processing: Adaptive cropping selects among multiple grids based on input-image shape and retains a global original-image version.
- Input Processing: Videos are represented by replacing video markers with multiple image placeholders corresponding to sampled frames.
4 Experiments
Experiments cover diverse visual question answering benchmarks and report strong results for mPLUG-Owl3 against similarly sized models. The evaluation emphasizes accuracy across knowledge, reasoning, accessibility, and text-reading settings.
- The visual question answering evaluation includes VQAv2, OK-VQA, GQA, VizWizQA, and TextVQA.These benchmarks span general VQA, external knowledge, reasoning, visually impaired users’ questions, and text-rich images.
- mPLUG-Owl3 outperforms 8B-level language models on VQAv2, OK-VQA, GQA, and VizWizQA.
- mPLUG-Owl3 surpasses the 32B-parameter EVLM3 on GQA and VizWizQA.
- In TextVQA, mPLUG-Owl3 is slightly below Idefics2 but exceeds other 8B models.
- Hyper Attention gives mPLUG-Owl3 higher inference speed and memory efficiency than same-scale models despite its 8B parameters.
4.2 General MLLM Benchmarks
mPLUG-Owl3 performs strongly across general single-image multimodal benchmarks, achieving state-of-the-art results on four of five evaluated tasks among 8B-level models, while lagging on text-rich AI2D inputs.
- The evaluated benchmarks cover bilingual multimodal capability, multimodal conversation, hallucination, and science-diagram understanding.MMBench-EN/CN assess Chinese and English contexts, MM-Vet uses GPT-4 evaluation, POPE measures multimodal hallucinations, and AI2D tests science-diagram understanding.
- mPLUG-Owl3 achieves state-of-the-art performance on MMBench-EN/CN, MM-Vet, and POPE across 8B-level models.It also matches or surpasses larger models such as CogVLM and EVLM-Chat.
- mPLUG-Owl3 does not achieve state-of-the-art performance on AI2D.The authors attribute this limitation to not fine-tuning the vision encoder, which restricts performance in text-rich scenarios.
4.3 Multi-image and Video Benchmark
mPLUG-Owl3 is competitive on short-video tasks and stronger on longer-video benchmarks, while also performing well across diverse multi-image reasoning and perception evaluations with a noted low-level-detail weakness.
- The evaluation spans video and multi-image benchmarks, including short and long videos, logical reasoning, mathematical and scientific tasks, low-level perception, and question answering.The benchmark captions report overall scores, with bold and underlining marking the highest and second-highest results.
- Video benchmarks: mPLUG-Owl3 significantly outperforms existing models on VideoMME and LongVideoBench, whose videos extend up to one hour.On shorter-video benchmarks, it achieves performance comparable to state-of-the-art models.
- Multi-image benchmarks: mPLUG-Owl3 surpasses existing models on NLVR2 and Mantis-Eval and significantly outperforms LLaVA-Interleave on MathVerse-mv and SciVerse-mv.These benchmarks assess multi-image logical, mathematical, and scientific reasoning.
- Multi-image benchmarks: mPLUG-Owl3 performs comparably to the state-of-the-art on Q-Bench2, which evaluates global low-level visual differences across multiple images.
- Multi-image benchmarks: mPLUG-Owl3 achieves state-of-the-art performance on MI-Bench tasks including General Comparison, Subtle Difference, Temporal Reasoning, Logical Reasoning, and Text-Rich Images.It also outperforms GPT-4V and GPT-4o on General Comparison.
4.4 Ablation Studies
Ablations show that Hyper Attention preserves single-image performance while generalizing better to multi-image and video settings than concatenation or alternative cross-attention placements.
- The ablation study uses LLaVA-1.5 training methods, Qwen1.5 7B, GQA and TextVQA, plus zero-shot multi-image and video evaluations.
- Attention structure: Hyper Attention achieves single-image performance comparable to direct feature concatenation while avoiding the degradation observed with Post-Cross-Attention.Pre-Cross-Attention performs better than Post-Cross-Attention but still incurs some performance loss.
- Attention structure: Concatenation can degrade multi-image performance because inadequate inter-image attention disrupts the model’s hidden states and textual instruction following.
- Attention structure: Hyper Attention uses the same text-conditioned cross-attention paradigm for single and multiple images, supporting better generalization from single-image training.
- Integration depth: Two Hyper Attention layers already provide strong single-image performance and generalization to video and multiple images, whereas eight layers do not improve single-image performance at this training scale.
- Component ablations: Adaptive gating and shared LayerNorm improve single-image understanding, while video performance improves even without inter-image positional encoding.For multiple images, contextual image position remains important; the ablation also examines MI-Rope.
4.5 Distractor Resistance in Long Visual Contexts
Distractor Resistance evaluates whether models remain accurate as irrelevant images accumulate, revealing substantial degradation for competing models in long visual contexts.
- Distractor Resistance addresses visual illusions caused by surrounding images, which multimodal needle-in-a-haystack evaluations do not detect.
- The evaluation inserts N−1 randomly selected MMBench dev-set images around the queried image, with N ranging from 1 to 400.CircularEval counts a question as correct only when all option-order and distractor variants are answered correctly.
- As distractor images increase, evaluation becomes significantly more challenging for the tested models.The comparison includes LLaVA-Next-Interleave 7B, Mantis-Idefics2, Qwen-VL, and mPLUG-Owl2.
- LLaVA-Next-Interleave accuracy drops to 43.18% at 20 images and 12.52% at 50 images.The model can handle approximately 20 images with 80GB of VRAM, extended to 50 using model parallelism, but not larger settings.
4.6 Qualitative Results
The qualitative examples show mPLUG-Owl3 handling multi-image dialogue, cross-image reasoning, and long-video questions, while also revealing errors under distractors and complex scenes.
- Multi-Image Understanding: mPLUG-Owl3 activates learned knowledge for cross-image reasoning and distinguishes multiple images using visual and cultural information.
- Multi-Image Understanding: The model finds differences between images from various views and describes correlations across them in multi-turn dialogue.
- Video Understanding: With 128 input frames, mPLUG-Owl3 captures key segments in videos longer than 40 minutes when answering detail-focused questions.LLaVA-Next-Interleave is evaluated with 8 frames in the comparison.
- Video Understanding: In a long multi-scene video, mPLUG-Owl3 understands temporal constraints and describes travel events in timeline order, but miscounts people after a later distracting scene.
- Video Understanding: The model also hallucinates by interpreting reefs as a beach and by transferring nighttime context to daytime boat activities.
5 Related Work
Related multimodal models differ in how they integrate visual features, trading off fine-grained information, token efficiency, context usage, and model interference. mPLUG-Owl3 instead targets efficient long-sequence multimodal interaction.
- Multimodal Large Language Models: MLLMs extend language models with visual encoders through image-text alignment, while newer applications require multi-image, retrieval, video, agent, and multi-document understanding.
- Multimodal Large Language Models: Directly concatenating visual features preserves fine-grained information but consumes many tokens, slowing training and inference and increasing memory use for long inputs.
- Multimodal Large Language Models: Query-based cross-attention compresses visual tokens to a fixed size, reducing token counts but suffering information loss.
- Multimodal Large Language Models: Flamingo-style cross-attention avoids occupying the language model’s context window but adds parameters, computational overhead, and possible interference with pretrained representations.
- Interleaved Multimodal Models: Video MLLMs treat temporally correlated frames as multiple images, while general multimodal dialogue requires in-context learning, cross-image reference, comparison, and reasoning.
- mPLUG-Owl3: mPLUG-Owl3 replaces visual-feature concatenation with efficient Hyper Attention to support multiple images, videos, and very long visual sequences with low resource overhead.
6 Conclusion
mPLUG-Owl3 advances single-image, multi-image, and video understanding with Hyper Attention that preserves fine-grained visual input while fusing visual and textual information. It also introduces Distractor Resistance and performs strongly on ultra-long visual sequences.
- mPLUG-Owl3 advances state-of-the-art performance on single-image, multi-image, and video tasks.
- Hyper Attention maintains fine-grained visual input and effectively fuses visual and textual information.
- Distractor Resistance evaluates whether models maintain focus during long visual sequences with distractions.
- mPLUG-Owl3 excels on ultra-long visual sequences and suggests a direction for efficient and effective multimodal language models.