Source-linked AI summary
MultiModal-GPT: A Vision and Language Model for Dialogue with Humans
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, Kai Chen
TL;DR
Multimodal assistants need to follow diverse vision-and-language instructions in human-oriented dialogue, while existing approaches can be computationally intensive or lack zero-shot multiturn dialogue. MultiModal-GPT fine-tunes OpenFlamingo with unified multimodal and language-only instruction data, and the authors report improved dialogue performance from joint training while identifying short-answer data as a problem.
Problem
Existing multimodal models can be computationally intensive with detailed visual information, and OpenFlamingo lacks zero-shot multiturn image-text dialogue.
Method
MultiModal-GPT jointly trains language-only and vision-language instructions using a unified template while parameter-efficiently fine-tuning OpenFlamingo with LoRA in the language decoder.
Results
Joint training of language-only and visual-language instructions effectively improves MultiModal-GPT’s dialogue performance, while short-answer datasets make responses overly brief.
Takeaways & Limitations
The model supports continuous human dialogue across multimodal tasks, including detailed captioning, object counting, and general question answering.
Abstract
from arXiv · showhide
We present a vision and language model named MultiModal-GPT to conduct multi-round dialogue with humans. MultiModal-GPT can follow various instructions from humans, such as generating a detailed caption, counting the number of interested objects, and answering general questions from users. MultiModal-GPT is parameter-efficiently fine-tuned from OpenFlamingo, with Low-rank Adapter (LoRA) added both in the cross-attention part and the self-attention part of the language model. We first construct instruction templates with vision and language data for multi-modality instruction tuning to make the model understand and follow human instructions. We find the quality of training data is vital for the dialogue performance, where few data containing short answers can lead the model to respond shortly to any instructions. To further enhance the ability to chat with humans of the MultiModal-GPT, we utilize language-only instruction-following data to train the MultiModal-GPT jointly. The joint training of language-only and visual-language instructions with the \emph{same} instruction template effectively improves dialogue performance. Various demos show the ability of continuous dialogue of MultiModal-GPT with humans. Code, dataset, and demo are at https://github.com/open-mmlab/Multimodal-GPT
1 Introduction
MultiModal-GPT is designed to follow multimodal instructions in human-oriented dialogue by extending OpenFlamingo with instruction tuning. The authors find that answer-length-sensitive data quality and joint language-only training affect dialogue performance.
- MultiModal-GPT targets versatile assistance across vision-and-language instructions, including detailed captioning, object counting, and general question answering.
- Earlier multimodal models aligned visual representations with LLM input spaces, but detailed or spatiotemporal visual information can impose substantial computational costs.
- Vicuna was refined from LLaMA using ChatGPT conversations, while omitting a language instruction-tuning phase in the described research.
- OpenFlamingo provides the foundation for fine-tuning a multimodal chatbot intended to support more human-like image-text conversations.
- Datasets with one- or two-word responses can make MultiModal-GPT generate similarly short answers across instructions, reducing user-friendliness.
- Jointly training language-only and visual-language instructions with a unified template effectively improves the model’s dialogue performance.
2 Unified Instruction Template
The paper uses a unified instruction format across language-only and vision-language data, adapting datasets to support instruction following and multimodal dialogue. It emphasizes response quality and consistent formatting during training.
- 2 Unified Instruction Template: A unified template integrates unimodal linguistic and multimodal vision-and-language data for joint MultiModal-GPT training.
- 2.1 Language-only Instruction Template: Language-only training inputs contain instruction, input, and response fields, with loss computed only on the response and <EOS> token.
- 2.1 Language-only Instruction Template: Dolly 15k and Alpaca GPT4 provide language-only instruction-following data formatted with a consistent prompt template.
- 2.2 Vision and Language Instruction Template: Vision-language inputs use question, response, and <image_token> fields, support multi-round dialogues when available, and calculate loss only on responses and <EOS>.
- 2.2 Vision and Language Instruction Template: Vision-language instruction data includes LLaVA, Mini-GPT4, A-OKVQA, COCO Caption, and OCR VQA across varied applications and domains.
- 2.2 Vision and Language Instruction Template: Standardized prompt formatting is used to help the model process vision-language information and respond in an instruction-following format.
- 2.2 Vision and Language Instruction Template: GPT-4 generates instructions for COCO Caption because that dataset mainly contains descriptive captions rather than instructional content.
3 Method
MultiModal-GPT combines OpenFlamingo’s visual processing components with a LLaMA language decoder and parameter-efficient joint instruction tuning. LoRA updates selected decoder modules while the base OpenFlamingo model remains frozen.
- 3.1 Architecture: The architecture uses a CLIP vision encoder, perceiver resampler, and LLaMA language decoder based on OpenFlamingo.
- 3.1 Architecture: Cross-attention conditions the language decoder on spatial features from the perceiver resampler to encode visual information into text.
- 3.2 Joint Training: Language-only and vision-language instruction-following data are jointly used to train MultiModal-GPT.
- 3.2 Joint Training: The whole OpenFlamingo model is frozen, while LoRA is added to self-attention, cross-attention, and FFN components in the language decoder.
- 3.2 Joint Training: Training predicts the next text token, with loss calculated only for response and <EOS> tokens.
4 Experiments
The experiments describe the training mixture and implementation, then demonstrate MultiModal-GPT’s dialogue and multimodal capabilities across recipes, recognition, counting, OCR, travel questions, and image reasoning.
- Training Data: The model jointly trains on language-only datasets and vision-language datasets to provide a diverse training environment.Language data include Dolly 15k and Alpaca GPT4; vision-language data include LLaVA, Mini-GPT4, A-OKVQA, COCO Caption, and OCR VQA.
- Implementation: Training uses 8 A100 GPUs for one epoch, with per-GPU batch size 1, gradient accumulation, and LoRA updates every 16 iterations.Each iteration contains one vision-language pair and one language-only instruction, yielding an aggregate batch size of 256.
- The Quality of Data Matters: The study excludes several short-answer vision-language datasets because they make the model generate responses of only one or two words.The excluded datasets are VQA v2.0, OKVQA, GQA, CLEVR, and NLVR.
- Demos: Demos show continuous dialogue in which the model identifies lasagna, gives a recipe, and then suggests places to eat.The example illustrates that the model can maintain an ongoing dialogue across related user requests.
- Demos: Additional demonstrations cover counting four women, recognizing book characters and film information, answering travel questions, and reasoning about image seasons.The experiments also report OCR capabilities and detailed image-description generation.