mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

• MLLMs extend LLMs with the ability to understand and handle visual problems.Source: §1 · pp. 1–2
• Fine-tuning LLMs for multimodal instruction can improve multimodal tasks while risking weaker text-task performance.Prior methods also restrict modality collaboration by mapping visual features into frozen LLMs through cross-modal alignment modules.Source: §1 · p. 2
• mPLUG-Owl2 uses a modularized network with shared functional modules and a modality-adaptive module to promote collaboration while preserving modality-specific features.Its language decoder serves as a universal interface for multimodal signals.Source: §1 · p. 2
• The two-stage training paradigm combines vision-language pre-training with joint vision-language instruction tuning, training the vision encoder to capture low- and high-level visual information.Source: §1 · p. 2
• mPLUG-Owl2 achieves state-of-the-art performance on 8 classic vision-language benchmarks with a single generic model and ranks first or second on 5 recent zero-shot multimodal benchmarks.Source: §1 · p. 2
• The model also achieves state-of-the-art results on multiple pure-text benchmarks, with modality collaboration improving text understanding, knowledge, and reasoning.Source: §1 · p. 2

• Flamingo aligns a frozen vision encoder with a language model through gated cross-attention, while PaLM-E inserts visual features into a pretrained language model through linear layers.Source: §2 · p. 2
• Directly integrating visual features can create lengthy visual sequences, motivating BLIP-2’s Q-former for reducing visual-token sequence length.Source: §2 · p. 2
• LLaVA, SVIT, and LRV-Instruction use image annotations and GPT-4 self-instruction to generate multimodal instructions and responses.Source: §2 · p. 3
• mPLUG-Owl and LLaVA-1.5 jointly train on language-only and vision-language instruction data to mitigate catastrophic forgetting.Source: §2 · p. 3

Model Architecture

• The model comprises a vision encoder, visual abstractor, text embedding layer, and language decoder that serves as a universal interface for vision-language tasks.The language decoder converts different vision-language tasks into text-generation tasks.Source: §3.2 · p. 3, §3.1 · p. 3
• The visual abstractor compresses visual token sequences before language-decoder processing, reducing computation from O((P + L)^2) to O((K + L)^2).This is especially beneficial when P ≫ K, including multiple-image settings and relatively short text sequences.Source: §3.2 · p. 3

Modality-Adaptive Module

• The Modality-Adaptive Module projects visual and language inputs into a shared semantic space while using modality-specific transformations to preserve their distinct features.It separates key and value projections, uses different layer normalization layers, and retains a shared query projection and shared FFN.Source: §3.3 · pp. 3–4

Training Paradigm

• Training first pre-trains visual-related modules and newly added language-decoder parameters, then jointly trains the entire model on language and multi-modal instructions.The instruction data is intended to improve visual-concept comprehension while preserving linguistic capabilities.Source: Figure 2 · p. 4, §3.4 · pp. 4–5

Modality-Adaptive Module

• The module separates modalities using indicators, normalizes them independently, and reformulates self-attention with shared queries but modality-specific key and value projections.This design addresses granularity mismatch and limits interference between visual and textual information while enabling collaboration.Source: §3.3 · pp. 3–4

Training Paradigm

• Evaluation uses image-captioning and visual-question-answering comparisons, alongside zero-shot multi-modal benchmarks, to assess the generic model.Table 1 reports CIDEr for captioning and accuracy for VQA, while Table 2 reports overall scores on five multi-modal benchmarks.Source: Table 1 · p. 5, Table 2 · p. 5

Experimental Setup

• mPLUG-Owl2 is pretrained on approximately 400 million image-text pairs and fine-tuned on mono-modal and multimodal instruction data.The instruction data covers captioning, question answering, region-aware QA, and language-only tasks.Source: §4.1 · p. 5

Multimodal Results

• The model surpasses previous generalist models on captioning and visual question answering, including 54.3% zero-shot accuracy on TextVQA.It achieves state-of-the-art performance on Flickr30K and shows strong zero-shot results on ScienceQA Image Set and VizWizQA.Source: §4.2 · p. 6

Pure-Text Results

• mPLUG-Owl2 improves pure-text performance, with gains of 2.3% on MMLU and 3.8% on BBH over comparison instruction-tuned LLMs.The evaluation covers MMLU, BBH, AGIEval, and ARC, using 5-shot MMLU and 0-shot evaluation for the other benchmarks.Source: §4.2 · p. 6, Table 3 · p. 6

Video Question Answering

• mPLUG-Owl2 achieves superior zero-shot results on MSRVTT-QA, MSVD-QA, and TGIF-QA under both exact-match and GPT-assisted evaluations.The GPT-assisted evaluation reports answer accuracy and relevance on a 1–5 relative score scale.Source: §4.2 · pp. 6–7, Table 4 · p. 6

Modality Collaboration

• Modality collaboration significantly improves examination and knowledge capabilities and also enhances text reasoning through visual concepts and relationships.The modality-adaptive module supports collaboration by preserving distinct roles for textual and visual information across processing stages.Source: §4.3 · p. 7, §4.4 · p. 8

Ablation Studies

• Using only multimodal instruction data lowers text-task performance by about 5.7%, whereas the modality-adaptive module improves both modalities by at least 0.6% on VQAv2 and 1.6% on MMLU.Trainable vision encoders improve VQAv2 and Q-Bench but reduce MM-Bench by 1.1%; layer-wise decay further increases TextVQA by 2.2%.Source: §4.3 · p. 7

• mPLUG-Owl2 leverages modality collaboration to enhance performance across both text and multi-modal tasks.Source: §5 · p. 8
• Shared functional modules promote modality collaboration while the modality-adaptive module preserves modality-specific characteristics.Source: §5 · p. 9
• Extensive evaluations show generalization across various tasks and state-of-the-art performance with a single generalized model.Source: §5 · p. 9
• The model is reported as the first MLLM to exhibit modality collaboration in both pure-text and multi-modal contexts.Source: §5 · p. 9
• The paper reports that modality collaboration improves language understanding, knowledge, and reasoning while supporting vision-language understanding.Source: §5 · p. 9

• MMHal-Bench evaluates image-description hallucination by comparing overall scores and hallucination rates across vision-language models.GPT-4 evaluates the overall score and hallucination rate.Source: §A.1 · p. 13, Figure 6 · p. 13
• mPLUG-Owl2 tends to generate responses with reduced hallucination compared with other methods.Source: §A.1 · p. 13
• In one example, mPLUG-Owl2 correctly identifies a white wall pattern that other models mistakenly perceive as a solid color.Source: §A.1 · p. 13
• In another example, mPLUG-Owl2 answers correctly and provides a more detailed explanation.Source: §A.1 · p. 13

• mPLUG-Owl2 achieves higher F1 scores on the popular and adversarial POPE splits than other MLLMs.The result is presented as evidence of robustness against object hallucination.Source: §A.2 · p. 13

MMBench

• MMBench comprehensively assesses diverse vision-language skills, with test-set results for various MLLMs presented in Table 10.Source: §A.3 · p. 13

MM-Vet

• Training mPLUG-Owl2’s visual encoder yields stronger OCR capability than LLaVA and Otter with the same backbone.Source: §A.4 · p. 13
• mPLUG-Owl2 surpasses LLaVA-13B on MM-Vet despite LLaVA-13B using a 13-billion-parameter language decoder.Source: §A.4 · p. 13

Q-Bench

• Training the visual encoder significantly improves mPLUG-Owl2’s low-level perception on Q-Bench.It outperforms a model with a stronger ViT-G visual encoder than its ViT-L encoder.Source: §A.5 · p. 13

MMHal-Bench

• The full MMHal-Bench evaluation results are provided in Table 13.Source: §A.6 · p. 13

Data processing

• Instruction-tuning data combine question-answer pairs sharing an image across several VQA datasets.The strategy is applied to VQAv2, GQA, OKVQA, and OCRVQA.Source: §B.1 · p. 13
• Multiple-choice A-OKVQA examples are augmented by switching option order to improve robustness.Source: §B.1 · p. 13

Qualitative comparison

• Figure 7 provides example-case comparisons with InstructBLIP, LLAVA, LLAVA-1.5, and MiniGPT-4.Source: Figure 7 · p. 14

Evaluation

• The supplementary evaluations include CircularEval results on MMBench and object-hallucination results using the POPE pipeline.The MMBench table reports multi-choice accuracy, while POPE uses “Yes” to indicate a positive response.Source: Table 10 · p. 15, Table 9 · p. 15

Data processing

• Regional-VQA datasets are included to improve regional abilities.Source: §B.1 · p. 15

Training

• Megatron model parallelism supports larger-resolution training while maintaining efficiency.Source: §B.2 · p. 15

Benchmark summary

• Table 16 summarizes mPLUG-Owl2’s evaluation benchmarks and their corresponding metrics.Source: C. Summary of the Evaluation Benchmarks · p. 15

Metrics

• The benchmark summary includes exact matching as an evaluation metric.EM is defined as exact matching.Source: C. Summary of the Evaluation Benchmarks · p. 15

Benchmark summary

• The evaluation resources are presented as a consolidated benchmark-and-metric reference.Source: C. Summary of the Evaluation Benchmarks · p. 15

Risks

• Using off-the-shelf LLMs and web-sourced data means mPLUG-Owl2 inherits weaknesses from those sources.Source: D. Broader Impact · p. 15
• Examples of these weaknesses include generating uncensored text or producing biased outputs.Source: D. Broader Impact · p. 15

Mitigations

• The authors enhance visual and instructional grounding and use joint vision-language instruction tuning on diverse high-quality datasets to address these shortcomings.Source: D. Broader Impact · p. 15

Deployment guidance

• They advise evaluating safety and fairness for each downstream application before deploying mPLUG-Owl2 models.Source: D. Broader Impact · p. 15