Source-linked AI summary
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, Fei Huang, Jingren Zhou
TL;DR
Existing LLMs have strong linguistic abilities but limited multimodal perception. mPLUG-Owl modularizes visual and language components and trains them in two stages, yielding strong multimodal instruction and dialogue performance with additional emerging abilities.
Problem
General LLMs demonstrate strong zero-shot language abilities but generally lack broad multimodal input and understanding.
Method
mPLUG-Owl modularizes a foundation LLM, visual knowledge module, and visual abstractor, using two-stage alignment and joint language-only and multimodal instruction tuning.
Results
mPLUG-Owl outperforms existing multimodal models on visual instruction understanding and multi-turn dialogue, with additional abilities including multi-image correlation and scene-text understanding.
Takeaways & Limitations
The model supports diverse unimodal and multimodal abilities through modality collaboration and shows potential for multimodal generation applications.
Takeaways & Limitations
Vision-only document comprehension remains limited: the model makes errors on some tasks and cannot provide usable responses for others.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated impressive zero-shot abilities on a variety of open-ended tasks, while recent research has also explored the use of LLMs for multi-modal generation. In this study, we introduce mPLUG-Owl, a novel training paradigm that equips LLMs with multi-modal abilities through modularized learning of foundation LLM, a visual knowledge module, and a visual abstractor module. This approach can support multiple modalities and facilitate diverse unimodal and multimodal abilities through modality collaboration. The training paradigm of mPLUG-Owl involves a two-stage method for aligning image and text, which learns visual knowledge with the assistance of LLM while maintaining and even improving the generation abilities of LLM. In the first stage, the visual knowledge module and abstractor module are trained with a frozen LLM module to align the image and text. In the second stage, language-only and multi-modal supervised datasets are used to jointly fine-tune a low-rank adaption (LoRA) module on LLM and the abstractor module by freezing the visual knowledge module. We carefully build a visually-related instruction evaluation set OwlEval. Experimental results show that our model outperforms existing multi-modal models, demonstrating mPLUG-Owl's impressive instruction and visual understanding ability, multi-turn conversation ability, and knowledge reasoning ability. Besides, we observe some unexpected and exciting abilities such as multi-image correlation and scene text understanding, which makes it possible to leverage it for harder real scenarios, such as vision-only document comprehension. Our code, pre-trained model, instruction-tuned models, and evaluation set are available at https://github.com/X-PLUG/mPLUG-Owl. The online demo is available at https://www.modelscope.cn/studios/damo/mPLUG-Owl.
1 Introduction
mPLUG-Owl addresses limited multimodal abilities in general LLMs with a modularized training paradigm and outperforms existing models on visual instruction tasks and dialogue.
- Current general LLMs generally lack broad multimodal input and ability, despite strong zero-shot linguistic performance.
- mPLUG-Owl combines a pretrained LLM, visual knowledge module, and visual abstractor in a modularized paradigm supporting multiple modalities.
- Its two-stage scheme first aligns images and text with a frozen LLM, then jointly fine-tunes LoRA on language and multimodal instructions while freezing visual knowledge.
- mPLUG-Owl outperforms existing models in multimodal instruction understanding and multi-turn dialogue, while exhibiting emerging abilities including multi-image correlation and scene-text understanding.
- OwlEval evaluates visual-related instruction capabilities across different models.
2 Related Work
Prior multimodal LLMs use varied combinations of frozen and trainable components but face alignment and capability limitations; mPLUG-Owl instead uses modular visual-language processing and staged instruction tuning.
- Large language models achieve strong zero-shot generalization but still struggle to perceive modalities beyond language, including vision and audio.
- End-to-end multimodal LLMs mainly differ in whether language and visual models are frozen or trainable across pretraining and instruction tuning.
- mPLUG-Owl encodes visual knowledge with a vision foundation model, language foundation model, and visual abstractor that summarizes image information into learnable tokens.
- Its training scheme addresses visual-language alignment through multimodal pretraining followed by joint instruction tuning with language-only and multimodal data.
4 Experiment
The experiments evaluate mPLUG-Owl against multimodal baselines on OwlEval, ablate its training strategy and instruction data, and examine qualitative capabilities. Results show strong overall, single-turn, multi-turn, knowledge-intensive QA, and reasoning performance, alongside a visual-information limitation in joke comprehension.
- Experimental Setup: OwlEval contains 82 open-ended questions based on 50 images, covering visual understanding, OCR, creation, knowledge-intensive QA, and referential interaction.Responses are manually rated from A to D.
- Quantitative Analysis: 66 A-or-B responses versus MiniGPT-4’s 54, while mPLUG-Owl received no D scores on OwlEval.The comparison excludes cases where MM-REACT failed to produce predictions.
- Quantitative Analysis: mPLUG-Owl performs strongly in both single-turn and multi-turn conversations across the reorganized OwlEval questions.The sets contain 50 single-turn responses and 52 multi-turn responses.
- Ablation Study: Joint multimodal pretraining and instruction tuning achieve the best ablation performance, whereas either component alone leaves instruction or visual-knowledge weaknesses.Without joint tuning, instruction understanding and generalization suffer; instruction tuning alone lacks visually related knowledge pretraining.
- Ablation Study: Text-only instruction tuning improves instruction understanding more, while multimodal instruction tuning improves knowledge and reasoning capabilities.Adding multimodal instruction data also improves performance on text-only tasks.
- Qualitative Analysis: mPLUG-Owl answers four of five movie characters in a knowledge-intensive QA case and provides the most accurate responses to referential image questions.It also reasons from lineup and tactics, although it incorrectly identifies “VGA” as “USB” in a visual joke case because of limited visual information in the training data.
5 Discussion and Limitation
mPLUG-Owl exhibits several emerging multimodal abilities, but performance remains uneven and limitations are evident across correlation, multilingual interaction, OCR, document comprehension, and practical creation.
- Multi-image Correlation: Multi-image correlation is emerging but not strong: the model can match a person across images yet fails on four-image relations and hallucinates text.
- Multilingual Conversation: Despite no multilingual training, mPLUG-Owl shows promising understanding in Chinese, French, and Japanese, but may not respond in the corresponding languages.
- Scene Text Understanding: Number perception remains limited, weakening OCR in simple scenes and subsequent reasoning and calculation in complex scenes.
- Vision-only Document Comprehension: Without document annotations, the model shows text recognition and document understanding, but produces errors and unusable responses for several applications.
- Open-ended Creation: Image-conditioned creation works well for poetry, lyrics, and advertisements, although more exploration is needed for functional and practical creations.
6 Conclusion
mPLUG-Owl enhances LLM multimodal abilities through modular learning and two-stage image-text alignment. The resulting model supports diverse unimodal and multimodal capabilities and shows potential for multimodal generation.
- mPLUG-Owl modularizes a foundation LLM, visual knowledge module, and visual abstractor to support multiple modalities and modality collaboration.
- Its two-stage image-text alignment learns visual knowledge with LLM assistance while maintaining and improving LLM generation abilities.
- Experimental results demonstrate impressive capabilities, indicating potential for various multimodal generation applications.
A Training Hyperparameters
The appendix reports training hyperparameters for mPLUG-Owl's multimodal pretraining and vision-language joint instruction-tuning stages.
- Table 4 reports training hyperparameters for the multimodal pretraining stage.
- Table 5 reports training hyperparameters for the vision-language joint instruction-tuning stage.
B Comparison with MM-REACT
The comparison section presents mPLUG-Owl examples spanning flowchart explanation, OCR, humor interpretation, and visual question answering. These examples include both qualitative explanations and extracted numerical answers.
- OCR comparison: The section includes OCR examples for simple scenes and two categories of complex scenes, alongside a comparison excluding unsuccessfully generated MM-REACT cases.The figure captions identify simple-scene OCR, complex-scene OCR, and the exclusion condition for the comparison.
- Flowchart understanding: mPLUG-Owl explains a flowchart for preparing an orange-and-milk drink through four ordered steps before serving it.The described process includes slicing oranges, blending them with milk, and serving the result in a glass.
- Scene text understanding: The model reads text in an image and explains its joke about appearance insecurity and pregnancy-test anxiety.The extracted message contrasts earlier and current feelings about appearance while referring to taking a pregnancy test.
- Numerical question answering: The examples include numerical visual question answering, such as identifying a total amount of $33.83.The answer directly reports the total amount shown in the image.
- Numerical question answering: For a fueling-cost question, the response reports an average total fueling cost of $10,000 and an average cost of $2.50 per gallon for 40,000 gallons.The response also lists vehicle-specific costs per gallon before presenting the aggregate calculation.