Source-linked AI summary
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, Mohamed Elhoseiny
TL;DR
The paper addresses the challenge of using one model to perform diverse vision-language tasks with simple multimodal instructions. MiniGPT-v2 introduces task-specific identifiers within a unified interface and achieves strong or state-of-the-art performance across visual question answering and visual grounding benchmarks, while still showing occasional hallucinations.
Problem
A single multimodal model must distinguish diverse vision-language tasks whose instructions can require different response formats.
Method
MiniGPT-v2 uses distinct task identifiers in a multi-task instruction template so one language model can differentiate and perform multiple vision-language tasks.
Results
MiniGPT-v2 achieves state-of-the-art or comparable performance across diverse benchmarks, including strong visual question answering and referring expression comprehension results.
Takeaways & Limitations
The model serves as a unified interface for varied vision-language inputs and can efficiently adapt to new vision-language tasks.
Takeaways & Limitations
MiniGPT-v2 occasionally hallucinates nonexistent objects or inaccurate grounded locations and underperforms specialist models in visual grounding.
Abstract
from arXiv · showhide
Large language models have shown their remarkable capabilities as a general interface for various language-related applications. Motivated by this, we target to build a unified interface for completing many vision-language tasks including image description, visual question answering, and visual grounding, among others. The challenge is to use a single model for performing diverse vision-language tasks effectively with simple multi-modal instructions. Towards this objective, we introduce MiniGPT-v2, a model that can be treated as a unified interface for better handling various vision-language tasks. We propose using unique identifiers for different tasks when training the model. These identifiers enable our model to better distinguish each task instruction effortlessly and also improve the model learning efficiency for each task. After the three-stage training, the experimental results show that MiniGPT-v2 achieves strong performance on many visual question-answering and visual grounding benchmarks compared to other vision-language generalist models. Our model and codes are available at https://minigpt-v2.github.io/
1 Introduction
MiniGPT-v2 targets a unified interface for diverse vision-language tasks, addressing ambiguity in how one instruction can require different response formats. It uses task identifiers, a simple visual-token architecture, and multi-stage training, achieving strong benchmark performance across tasks.
- Vision-language models support applications including visual assistance, image captioning, visual question answering, and referring expression comprehension.
- A single instruction can require different outputs, such as a bounding box, natural-language location, or multiple detected locations, depending on the task.
- MiniGPT-v2 uses six unique task-identifier tokens to distinguish tasks during multi-modal instruction training.For example, visual question-answering samples use a [vqa] identifier token.
- The model directly projects ViT visual tokens into a large language model’s feature space and uses 448x448 training images.Concatenating every four neighboring visual tokens reduces their total number by 75%, and training uses three stages.
- MiniGPT-v2 achieves SOTA or comparable performance across image captioning, visual question answering, and visual grounding benchmarks against vision-language generalist models.On VSR, it outperforms MiniGPT-4 by 21.3%, InstructBLIP by 11.3%, and LLaVA by 11.7%.
2 Related Work
Related work traces the development of large language models from foundation models to instruction-following systems, alongside efforts to align them with visual inputs. This progression includes multimodal models for captioning and visual question answering, culminating in broader vision-language capabilities.
- Advanced Large Language Models (LLMs): Early foundation models such as GPT-2 and BERT were trained on web-scale text datasets and marked a breakthrough in natural language processing.
- Advanced Large Language Models (LLMs): Later large language models increased model capacity and training data, while InstructGPT and ChatGPT refined instruction following and complex task performance.
- Advanced Large Language Models (LLMs): LLaMA-based models such as Alpaca, Vicuna, and MPT were fine-tuned with additional instruction datasets to improve human instruction following.
- Visual Aligning with LLMs: VisualGPT and Frozen extended pre-trained language models to image captioning and visual question answering by aligning visual inputs with language models.
- Visual Aligning with LLMs: This early multimodal exploration preceded subsequent vision-language research including Flamingo and BLIP-2.
3 Method
MiniGPT-v2 combines a frozen visual backbone, token-compressing projection layer, and LLaMA-2 language model with task-oriented instructions. Its three-stage training uses task identifiers and progressively broader, finer-grained, and conversational multimodal data.
- 3.1 Model Architecture: MiniGPT-v2 has three components: a visual backbone, a linear projection layer, and a large language model.
- 3.1 Model Architecture: The visual backbone remains frozen, processes 448x448 images, and uses interpolated positional encoding for the higher resolution.
- 3.1 Model Architecture: Concatenating four adjacent visual tokens before projection reduces the number of visual input tokens by 4 times, improving high-resolution efficiency.
- 3.1 Model Architecture: LLaMA2-chat (7B) serves as a unified interface, generating textual bounding-box representations for visual grounding tasks.
- 3.2 Multi-task Instruction Template: The multimodal template combines image features, a task identifier token, and the instruction input within the LLaMA-2 conversation format.
- 3.2 Multi-task Instruction Template: Six task identifiers distinguish visual question answering, image captioning, grounded captioning, REC, REG, and object parsing and grounding.
- 3.3 Multi-task Instruction Training: Three-stage training progresses from broad weakly labeled and fine-grained vision-language data, through fine-grained multitask training, to multimodal instruction tuning.
- 3.3 Multi-task Instruction Training: The third stage adds LLaVA, Flickr30k, a constructed multitask dataset, and Unnatural Instruction while retaining second-stage datasets.
4 Experiments
MiniGPT-v2 is evaluated across VQA, visual grounding, captioning, and hallucination settings, with strong benchmark performance and qualitative multi-task capabilities. Task identifiers improve average VQA accuracy, while hallucinations and specialist-model gaps remain limitations.
- Experiments cover detailed and grounded captioning, VQA, visual grounding, and referring expression comprehension across diverse benchmarks.
- MiniGPT-v2 achieves SOTA or comparable performance against vision-language generalist models across diverse tasks.
- 84.29% average accuracy versus 83.96% for Shikra (13B) demonstrates MiniGPT-v2’s strong referring expression comprehension performance.
- 1.2% average top-1 accuracy improvement shows that task-oriented instruction training benefits VQA performance.
- Task identifiers support varied image descriptions with lower hallucination than the reported baselines across three instructional variants.
- Qualitative examples show object identification, grounded captioning, VQA, referring expression comprehension, and text-based object parsing and grounding.
- The model occasionally hallucinates nonexistent objects or inaccurate grounded locations, and it underperforms specialist models in visual grounding.
5 Conclusion
The conclusion presents MiniGPT-v2 as a unified interface for vision-language multi-task learning. Distinct task identifiers help separate tasks and improve learning efficiency, while results and adaptation experiments support broad capability.
- MiniGPT-v2 uses distinct identifiers during training and inference to differentiate vision-language tasks and improve learning efficiency.
- The model achieves state-of-the-art results across many VQA and referring expression comprehension benchmarks.
- MiniGPT-v2 efficiently adapts to new vision-language tasks, indicating potential applications in the vision-language community.
A Appendix
The supplementary material provides additional qualitative outputs illustrating MiniGPT-v2’s vision-language multi-tasking capabilities.
- The supplementary material contains more qualitative results generated by MiniGPT-v2.
A.1 Instruction template for various vision-language tasks
The appendix specifies task-oriented instruction templates for VQA and visual grounding benchmarks. Templates use task identifiers and task-specific response formats.
- RefCOCO, RefCOCO+, and RefCOCOg use the [refer] identifier with a location request for referring expressions.
- VizWiz uses [vqa] and permits the response “unanswerable” when image information is insufficient.
- Hateful Meme prompts use [vqa] to ask whether the image text is hateful.
- VSR uses [vqa] to ask whether an image-grounded statement is true or false.
- IconQA, GQA, and OKVQA use [vqa] prompts requesting a single word or phrase.
A.2 Additional Qualitative Results
The qualitative results present four examples each across grounded captioning, object parsing and grounding, referring expression comprehension, and object identification. MiniGPT-v2 follows task identifiers across new scenes, but some examples still show inaccurate grounding and hallucination.
- Qualitative task coverage: Four examples each cover grounded image captioning, object parsing and grounding, referring expression comprehension, and object identification.The figures span Figures 4–19 across these four task categories.
- Grounded image captioning: Grounded captioning examples provide detailed spatial locations for objects.
- Object identification: Object identification examples generate the expected object names.
- Task-oriented interaction: MiniGPT-v2 understands new scenes and follows the question identifier to produce task-specific responses.
- Observed limitations: Several persons are not grounded accurately in Fig. 6, and Fig. 7 hallucinates a vase absent from the image.