Source-linked AI summary
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny
TL;DR
Existing vision-language models rarely match GPT-4’s advanced multimodal abilities, whose technical basis remains undisclosed. MiniGPT-4 aligns visual features with the frozen Vicuna LLM and demonstrates comparable capabilities, including detailed descriptions and website creation from drafts.
Problem
GPT-4’s exceptional vision-language abilities remain technically unexplained, motivating investigation into whether advanced LLMs underlie such multimodal capabilities.
Method
MiniGPT-4 aligns a frozen visual encoder with the frozen Vicuna LLM through one projection layer, followed by second-stage finetuning on detailed image descriptions.
Results
MiniGPT-4 demonstrates advanced multimodal abilities including detailed image descriptions, website creation from handwritten drafts, recipes, advertisements, and image-inspired poems.
Takeaways & Limitations
Properly aligning visual features with an advanced LLM can produce vision-language capabilities comparable to those demonstrated by GPT-4.
Takeaways & Limitations
Detailed image descriptions remain susceptible to hallucination, with longer captions showing higher hallucination rates.
Abstract
from arXiv · showhide
The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. However, the technical details behind GPT-4 continue to remain undisclosed. We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts. Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g., repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability. Our code, pre-trained model, and collected dataset are available at https://minigpt-4.github.io/.
1 INTRODUCTION
MiniGPT-4 tests whether aligning visual features with an advanced LLM can produce GPT-4-like vision-language abilities. It uses a largely frozen vision-language stack and addresses unreliable generation through detailed image-description finetuning.
- Motivation and contribution: MiniGPT-4 aligns visual features with Vicuna to investigate whether advanced LLMs enable GPT-4-like vision-language capabilities.The model is presented as evidence that this alignment can yield advanced capabilities comparable to GPT-4 demonstrations.
- Architecture: Its architecture combines a pretrained ViT and Q-Former vision encoder, one linear projection layer, and the Vicuna language model.Only the projection layer is trained while the other vision and language components remain frozen.
- Emergent capabilities: MiniGPT-4 generates intricate image descriptions, creates websites from handwritten instructions, and explains unusual visual phenomena.It also produces detailed cooking recipes from food images and exhibits other abilities beyond the cited GPT-4 demonstrations.
- Training limitation: Short image-caption training alone causes unnatural language generation, including repetition and fragmentation.The limitation is attributed to insufficiently detailed training pairs.
- Training limitation: Finetuning with a small, detailed image-description dataset significantly improves MiniGPT-4’s generation reliability and usability.This second-stage finetuning addresses the unnatural outputs produced after initial alignment.
2 RELATED WORKS
Related work traces progress from increasingly large language models to their use as decoders or collaborators in vision-language systems. These approaches leverage language-model knowledge and cross-modal transfer to improve multimodal task performance.
- Large Language Models: Scaling training data and parameter counts drove progress from early models such as BERT, GPT-2, and T5 to GPT-3, which used 175 billion parameters.The passage describes GPT-3 as achieving significant breakthroughs across numerous language benchmarks.
- Leveraging Pre-trained LLMs in Vision-Language Tasks: Recent vision-language research increasingly uses autoregressive language models as decoders, enabling cross-modal transfer between language and multimodal domains.The passage identifies this decoder-based approach as a growing trend in vision-language tasks.
- Leveraging Pre-trained LLMs in Vision-Language Tasks: Other systems use ChatGPT to coordinate specialized visual foundation models or generate questions, as illustrated by Visual ChatGPT, MM-REACT, and ChatCaptioner.These roles support collaboration among diverse models and more complex vision-language challenges.
3 METHOD
MiniGPT-4 aligns a frozen BLIP-2 visual encoder and frozen Vicuna language model through a single linear projection layer. It uses two-stage training: broad image-text pretraining followed by high-quality conversational fine-tuning to improve output reliability and usability.
- Model architecture: MiniGPT-4 uses Vicuna as its language decoder and the BLIP-2 visual encoder, connecting them through a single linear projection layer.Vicuna is built on LLaMA, while the visual encoder combines a ViT backbone with a pretrained Q-Former.
- Training strategy: The model is trained in two stages: large-scale aligned image-text pretraining followed by fine-tuning on a smaller, high-quality conversational dataset.The second stage is designed to enhance generation reliability and usability.
- Initial pretraining: 20,000 training steps with batch size 256 cover approximately 5 million image-text pairs while keeping the pretrained vision encoder and LLM frozen.Only the linear projection layer is pretrained, using Conceptual Caption, SBU, and LAION data.
- Issues after pretraining: The first-stage model can produce incoherent outputs, including repetitions, fragmented sentences, and irrelevant content, limiting fluent visual conversation.These issues motivate a second-stage alignment process and a curated detailed image-description dataset.
- Second-stage alignment: 400 training steps with batch size 12 take around 7 minutes on one A100 GPU and yield more natural and reliable language outputs.The fine-tuning data are generated from image descriptions, post-processed, manually verified, and filtered for quality.
4 EXPERIMENTS
Experiments show that MiniGPT-4 exhibits diverse advanced vision-language abilities and outperforms BLIP-2 on detailed captioning. Second-stage finetuning improves generation fluency, while hallucination and spatial-understanding limitations remain.
- Qualitative abilities: MiniGPT-4 demonstrates detailed image description, meme interpretation, recipe generation, advertisement creation, website drafting, plant-disease diagnosis, and image-inspired poetry.Qualitative comparisons include eight examples against BLIP-2, highlighting distinct capabilities.
- Advanced Abilities: 65% of requests receive successful responses in the advanced vision-language evaluation spanning meme interpretation, recipe generation, advertisement creation, and poem composition.The evaluation contains 100 diverse images across four tasks.
- Image Captioning: 66.2% success rate makes MiniGPT-4 more accurate than BLIP-2’s 27.5% on captions covering ground-truth visual objects and relationships.ChatGPT judges whether generated captions cover the visual information in ground-truth captions.
- Effectiveness of the second-stage finetuning: Second-stage finetuning largely mitigates repetitive, fragmented, irrelevant, and incomplete outputs, producing complete and fluent captions.The impact is measured on detailed description and poem-writing tasks using 100 COCO test images and manually counted failures.
- Second stage with Localized Narratives: Replacing the second-stage dataset with Localized Narratives enables long descriptions but yields more monotonous expressions and weaker generalization on complex tasks such as meme explanation.The performance gap may result from monotonous and repeated descriptions in Localized Narratives.
- Limitations: Longer captions hallucinate more: MiniGPT-4 averages 175 words with a higher hallucination rate, whereas its 28.8-word short captions have a lower rate.BLIP-2 averages 6.5 words, hallucinates less, but covers fewer objects; spatial localization also remains limited.
5 DISCUSSION
MiniGPT-4’s advanced vision-language abilities are framed as compositional skills combining image understanding with language generation. Its second-stage finetuning addresses the mismatch between image-caption language and modern LLM generation that otherwise distorts outputs and hinders compositional generalization.
- Compositional skills: Advanced vision-language abilities can be understood as compositional skills rooted in image understanding and language generation.Image-based poem writing illustrates this composition: advanced LLMs can craft poems from instructions, while image understanding enables generalization to image-based tasks.
- Training stages: During pretraining, MiniGPT-4 learns image understanding by modeling correlations between images and short descriptions from image-caption datasets.These datasets use a language style that differs from modern LLM generation.
- Training stages: Second-stage finetuning restores language generation ability because caption-style language causes distorted outputs and hinders successful compositional generalization.The discussion identifies language-style mismatch as the reason for introducing the second training stage.
A APPENDIX · A.1 MORE QUALITATIVE RESULTS
The appendix presents qualitative examples of MiniGPT-4’s capabilities across advertisement promotion, website creation, factual retrieval, poem writing, food recipe generation, and plant cultivating.
- A.1 MORE QUALITATIVE RESULTS: The appendix includes a qualitative example of advertisement promotion.
- A.1 MORE QUALITATIVE RESULTS: A separate example demonstrates website creation.
- A.1 MORE QUALITATIVE RESULTS: The qualitative results also include factual retrieval.
- A.1 MORE QUALITATIVE RESULTS: Another example showcases poem writing.
- A.1 MORE QUALITATIVE RESULTS: The appendix presents an example of food recipe generation.
- A.1 MORE QUALITATIVE RESULTS: It also includes a plant cultivating example.
A.2 EVALUATION IN TRADITIONAL VQA BENCHMARKS
MiniGPT-4 evaluates whether a minimally designed vision-language model can replicate GPT-4-like multimodal capabilities. Its methodology restricts learnable capacity to one linear layer and uses 5 million training pairs, contrasting with BLIP-2’s 129 million.
- The evaluation aims to replicate GPT-4-like capabilities, including detailed image descriptions and website creation from hand-drawn drafts.
- MiniGPT-4 intentionally uses a minimal methodology to emphasize the most crucial component of advanced vision-language skills.
- The model has only one learnable linear layer and is trained with 5 million pairs, versus BLIP-2’s 129 million.
A.3 DETAILS OF CAPTION EVALUATION
Caption evaluation uses ChatGPT to judge whether comparison captions cover the objects and visual relations in reference captions. For COCO, one ground-truth caption is randomly selected as the reference, and ChatGPT returns only yes or no.
- Evaluation procedure: ChatGPT evaluates whether baseline-model captions cover all objects and visual relations in the ground-truth captions.The evaluation uses a prompted comparison between a reference caption and a comparison caption.
- Evaluation procedure: For COCO, one ground-truth caption is randomly selected and treated as the reference caption.The comparison caption is assessed against this reference.
- Evaluation prompt: The evaluation prompt asks whether caption2 covers every object and visual relation shown in caption1 and requires only a yes-or-no answer.The prompt includes a ground-truth caption as caption1 and a comparison caption as caption2, without allowing explanations.
A.4 MORE QUALITATIVE ABLATION RESULTS
This section presents qualitative ablation results for recipe generation and detailed image description.
- The study includes an ablation analysis of recipe generation.
- Together, the figures cover qualitative ablations across recipe generation and detailed description.
- A separate ablation analysis examines detailed description.