Source-linked AI summary
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, Jingren Zhou
TL;DR
Prior MLLMs have struggled to balance multimodal gains with preserved text capabilities because modality interference can limit collaboration. mPLUG-Owl2 uses a modular decoder with shared and modality-adaptive components, and experiments show strong generalization across multimodal and pure-text tasks with one generic model.
Problem
Existing MLLM strategies restrict modality collaboration or risk weakening text-task performance when improving multimodal capabilities.
Method
mPLUG-Owl2 uses a modularized language decoder with shared functional modules for collaboration and modality-adaptive modules that preserve modality-specific features.
Results
mPLUG-Owl2 achieves state-of-the-art performance across 8 classic vision-language benchmarks and multiple pure-text benchmarks using a single generic model.
Takeaways & Limitations
mPLUG-Owl2 demonstrates modality collaboration in both pure-text and multimodal contexts while improving language understanding, knowledge, and reasoning.
Abstract
from arXiv · showhide
Multi-modal Large Language Models (MLLMs) have demonstrated impressive instruction abilities across various open-ended tasks. However, previous methods primarily focus on enhancing multi-modal capabilities. In this work, we introduce a versatile multi-modal large language model, mPLUG-Owl2, which effectively leverages modality collaboration to improve performance in both text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design, with the language decoder acting as a universal interface for managing different modalities. Specifically, mPLUG-Owl2 incorporates shared functional modules to facilitate modality collaboration and introduces a modality-adaptive module that preserves modality-specific features. Extensive experiments reveal that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal tasks and achieving state-of-the-art performances with a single generic model. Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality collaboration phenomenon in both pure-text and multi-modal scenarios, setting a pioneering path in the development of future multi-modal foundation models.
1. Introduction
MLLMs seek to extend language models with visual capabilities, but balancing modality collaboration against interference remains difficult. mPLUG-Owl2 addresses this with modular modality handling and achieves strong generalization across vision-language and text tasks.
- MLLMs extend LLMs with the ability to understand and handle visual problems.
- Fine-tuning LLMs for multimodal instruction can improve multimodal tasks while risking weaker text-task performance.Prior methods also restrict modality collaboration by mapping visual features into frozen LLMs through cross-modal alignment modules.
- mPLUG-Owl2 uses a modularized network with shared functional modules and a modality-adaptive module to promote collaboration while preserving modality-specific features.Its language decoder serves as a universal interface for multimodal signals.
- The two-stage training paradigm combines vision-language pre-training with joint vision-language instruction tuning, training the vision encoder to capture low- and high-level visual information.
- mPLUG-Owl2 achieves state-of-the-art performance on 8 classic vision-language benchmarks with a single generic model and ranks first or second on 5 recent zero-shot multimodal benchmarks.
- The model also achieves state-of-the-art results on multiple pure-text benchmarks, with modality collaboration improving text understanding, knowledge, and reasoning.
2. Related Work
Related work develops unified vision-language models by augmenting language models with visual features through several alignment and instruction-generation strategies. These approaches support zero-shot generalization but retain limitations involving visual-sequence length and catastrophic forgetting.
- Flamingo aligns a frozen vision encoder with a language model through gated cross-attention, while PaLM-E inserts visual features into a pretrained language model through linear layers.
- Directly integrating visual features can create lengthy visual sequences, motivating BLIP-2’s Q-former for reducing visual-token sequence length.
- LLaVA, SVIT, and LRV-Instruction use image annotations and GPT-4 self-instruction to generate multimodal instructions and responses.
- mPLUG-Owl and LLaVA-1.5 jointly train on language-only and vision-language instruction data to mitigate catastrophic forgetting.
3. Methodology
mPLUG-Owl2 uses a modular architecture and two-stage training to process visual and textual inputs through a shared language-decoder interface. Its modality-adaptive module promotes collaboration in a shared semantic space while preserving modality-specific features.
- Model Architecture: The model comprises a vision encoder, visual abstractor, text embedding layer, and language decoder that serves as a universal interface for vision-language tasks.The language decoder converts different vision-language tasks into text-generation tasks.
- Model Architecture: The visual abstractor compresses visual token sequences before language-decoder processing, reducing computation from O((P + L)^2) to O((K + L)^2).This is especially beneficial when P ≫ K, including multiple-image settings and relatively short text sequences.
- Modality-Adaptive Module: The Modality-Adaptive Module projects visual and language inputs into a shared semantic space while using modality-specific transformations to preserve their distinct features.It separates key and value projections, uses different layer normalization layers, and retains a shared query projection and shared FFN.
- Training Paradigm: Training first pre-trains visual-related modules and newly added language-decoder parameters, then jointly trains the entire model on language and multi-modal instructions.The instruction data is intended to improve visual-concept comprehension while preserving linguistic capabilities.
- Modality-Adaptive Module: The module separates modalities using indicators, normalizes them independently, and reformulates self-attention with shared queries but modality-specific key and value projections.This design addresses granularity mismatch and limits interference between visual and textual information while enabling collaboration.
- Training Paradigm: Evaluation uses image-captioning and visual-question-answering comparisons, alongside zero-shot multi-modal benchmarks, to assess the generic model.Table 1 reports CIDEr for captioning and accuracy for VQA, while Table 2 reports overall scores on five multi-modal benchmarks.
4. Experiments
Experiments evaluate mPLUG-Owl2 across multimodal, pure-text, video, and modality-collaboration settings. The model shows strong generalization, while ablations identify benefits and trade-offs from joint tuning, trainable vision encoders, resolution, and modality-adaptive processing.
- Experimental Setup: mPLUG-Owl2 is pretrained on approximately 400 million image-text pairs and fine-tuned on mono-modal and multimodal instruction data.The instruction data covers captioning, question answering, region-aware QA, and language-only tasks.
- Multimodal Results: The model surpasses previous generalist models on captioning and visual question answering, including 54.3% zero-shot accuracy on TextVQA.It achieves state-of-the-art performance on Flickr30K and shows strong zero-shot results on ScienceQA Image Set and VizWizQA.
- Pure-Text Results: mPLUG-Owl2 improves pure-text performance, with gains of 2.3% on MMLU and 3.8% on BBH over comparison instruction-tuned LLMs.The evaluation covers MMLU, BBH, AGIEval, and ARC, using 5-shot MMLU and 0-shot evaluation for the other benchmarks.
- Video Question Answering: mPLUG-Owl2 achieves superior zero-shot results on MSRVTT-QA, MSVD-QA, and TGIF-QA under both exact-match and GPT-assisted evaluations.The GPT-assisted evaluation reports answer accuracy and relevance on a 1–5 relative score scale.
- Modality Collaboration: Modality collaboration significantly improves examination and knowledge capabilities and also enhances text reasoning through visual concepts and relationships.The modality-adaptive module supports collaboration by preserving distinct roles for textual and visual information across processing stages.
- Ablation Studies: Using only multimodal instruction data lowers text-task performance by about 5.7%, whereas the modality-adaptive module improves both modalities by at least 0.6% on VQAv2 and 1.6% on MMLU.Trainable vision encoders improve VQAv2 and Q-Bench but reduce MM-Bench by 1.1%; layer-wise decay further increases TextVQA by 2.2%.
5. Conclusion
mPLUG-Owl2 uses modality collaboration to generalize across text and multi-modal tasks with one generalized model. Its analyses and ablations support the role of modality-adaptive design, although the supplied passages provide no substantive new experiment here.
- mPLUG-Owl2 leverages modality collaboration to enhance performance across both text and multi-modal tasks.
- Shared functional modules promote modality collaboration while the modality-adaptive module preserves modality-specific characteristics.
- Extensive evaluations show generalization across various tasks and state-of-the-art performance with a single generalized model.
- The model is reported as the first MLLM to exhibit modality collaboration in both pure-text and multi-modal contexts.
- The paper reports that modality collaboration improves language understanding, knowledge, and reasoning while supporting vision-language understanding.
A.1. Hallucination Evaluation
The hallucination evaluation compares mPLUG-Owl2 with recent vision-language models using MMHal-Bench and qualitative examples. The reported results indicate reduced hallucination and more accurate, detailed responses.
- MMHal-Bench evaluates image-description hallucination by comparing overall scores and hallucination rates across vision-language models.GPT-4 evaluates the overall score and hallucination rate.
- mPLUG-Owl2 tends to generate responses with reduced hallucination compared with other methods.
- In one example, mPLUG-Owl2 correctly identifies a white wall pattern that other models mistakenly perceive as a solid color.
- In another example, mPLUG-Owl2 answers correctly and provides a more detailed explanation.
A.2. POPE Evaluation
The POPE evaluation measures object hallucination robustness through F1 scores on popular and adversarial splits. mPLUG-Owl2 achieves higher F1 scores on both splits than other MLLMs.
- mPLUG-Owl2 achieves higher F1 scores on the popular and adversarial POPE splits than other MLLMs.The result is presented as evidence of robustness against object hallucination.
A.3. Detailed Evaluation Results on MMBench
The supplied passages describe detailed evaluations on MMBench, MM-Vet, Q-Bench, and MMHal-Bench. They report stronger OCR and low-level visual perception under the stated training setup, while also pointing to full MMHal-Bench results.
- MMBench: MMBench comprehensively assesses diverse vision-language skills, with test-set results for various MLLMs presented in Table 10.
- MM-Vet: Training mPLUG-Owl2’s visual encoder yields stronger OCR capability than LLaVA and Otter with the same backbone.
- MM-Vet: mPLUG-Owl2 surpasses LLaVA-13B on MM-Vet despite LLaVA-13B using a 13-billion-parameter language decoder.
- Q-Bench: Training the visual encoder significantly improves mPLUG-Owl2’s low-level perception on Q-Bench.It outperforms a model with a stronger ViT-G visual encoder than its ViT-L encoder.
- MMHal-Bench: The full MMHal-Bench evaluation results are provided in Table 13.
B.1. Data Mixture
The supplementary material documents mPLUG-Owl2’s instruction-tuning data processing, training setup, and evaluation resources, alongside benchmark tables and qualitative comparisons.
- Data processing: Instruction-tuning data combine question-answer pairs sharing an image across several VQA datasets.The strategy is applied to VQAv2, GQA, OKVQA, and OCRVQA.
- Data processing: Multiple-choice A-OKVQA examples are augmented by switching option order to improve robustness.
- Qualitative comparison: Figure 7 provides example-case comparisons with InstructBLIP, LLAVA, LLAVA-1.5, and MiniGPT-4.
- Evaluation: The supplementary evaluations include CircularEval results on MMBench and object-hallucination results using the POPE pipeline.The MMBench table reports multi-choice accuracy, while POPE uses “Yes” to indicate a positive response.
- Data processing: Regional-VQA datasets are included to improve regional abilities.
- Training: Megatron model parallelism supports larger-resolution training while maintaining efficiency.
C. Summary of the Evaluation Benchmarks
The evaluation-benchmark summary identifies the benchmarks used for mPLUG-Owl2 and records exact matching as one evaluation metric.
- Benchmark summary: Table 16 summarizes mPLUG-Owl2’s evaluation benchmarks and their corresponding metrics.
- Metrics: The benchmark summary includes exact matching as an evaluation metric.EM is defined as exact matching.
- Benchmark summary: The evaluation resources are presented as a consolidated benchmark-and-metric reference.
D. Broader Impact
The broader-impact discussion identifies inherited weaknesses from off-the-shelf language models and web-sourced data, including uncensored or biased outputs, and advises application-specific safety evaluation.
- Risks: Using off-the-shelf LLMs and web-sourced data means mPLUG-Owl2 inherits weaknesses from those sources.
- Risks: Examples of these weaknesses include generating uncensored text or producing biased outputs.
- Mitigations: The authors enhance visual and instructional grounding and use joint vision-language instruction tuning on diverse high-quality datasets to address these shortcomings.
- Deployment guidance: They advise evaluating safety and fairness for each downstream application before deploying mPLUG-Owl2 models.