Source-linked AI summary
Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, Jiaya Jia
TL;DR
Mini-Gemini addresses the performance gap between academic vision-language models and advanced systems by combining high-resolution visual tokens, high-quality data, and expanded multimodal workflows. The framework achieves leading zero-shot benchmark performance, surpassing several private models, while counting and complex visual reasoning remain limited.
Problem
Academic vision-language models still lag behind advanced systems such as GPT-4 and Gemini, while higher-resolution processing increases computational demands.
Method
Mini-Gemini combines dual-encoder high-resolution visual refinement, high-quality data construction, and reasoning-based any-to-any image-text workflows.
Results
Mini-Gemini achieves leading performance across zero-shot benchmarks and surpasses private models including Gemini Pro, Qwen-VL-Plus, and GPT-4V on challenging datasets.
Takeaways & Limitations
The framework supports dense and mixture-of-experts language models from 2B to 34B parameters while combining image understanding, reasoning, and generation.
Takeaways & Limitations
Counting and complex visual reasoning remain unsatisfactory, potentially because corresponding pretraining data are lacking.
Abstract
from arXiv · showhide
In this work, we introduce Mini-Gemini, a simple and effective framework enhancing multi-modality Vision Language Models (VLMs). Despite the advancements in VLMs facilitating basic visual dialog and reasoning, a performance gap persists compared to advanced models like GPT-4 and Gemini. We try to narrow the gap by mining the potential of VLMs for better performance and any-to-any workflow from three aspects, i.e., high-resolution visual tokens, high-quality data, and VLM-guided generation. To enhance visual tokens, we propose to utilize an additional visual encoder for high-resolution refinement without increasing the visual token count. We further construct a high-quality dataset that promotes precise image comprehension and reasoning-based generation, expanding the operational scope of current VLMs. In general, Mini-Gemini further mines the potential of VLMs and empowers current frameworks with image understanding, reasoning, and generation simultaneously. Mini-Gemini supports a series of dense and MoE Large Language Models (LLMs) from 2B to 34B. It is demonstrated to achieve leading performance in several zero-shot benchmarks and even surpasses the developed private models. Code and models are available at https://github.com/dvlab-research/MiniGemini.
1 Introduction
Mini-Gemini addresses the gap between academic VLMs and advanced private models by combining efficient high-resolution processing, higher-quality data, and expanded multimodal applications. Its framework uses dual visual encoders and achieves leading results across challenging multimodal benchmarks.
- Motivation: VLMs still lag behind models such as GPT-4 and Gemini despite progress in visual dialog and reasoning.The paper identifies data, computational resources, and model capabilities as central differences.
- Motivation: Higher image resolution improves visual embeddings but increases visual-token counts, computation, and cost, especially for multiple images.The paper frames this efficiency-quality trade-off as a central design problem.
- Approach: Mini-Gemini combines efficient high-resolution candidates, high-quality datasets, and expanded applications to improve VLMs under manageable resource constraints.The framework maintains the visual token count while enriching visual detail and expands training data and functionality.
- Approach: A dual-encoder pipeline uses low-resolution visual queries with high-resolution candidate keys and values for visual-token enhancement.The low-resolution encoder provides queries, while the high-resolution encoder supplies reference features during attention.
- Results: Mini-Gemini supports LLMs from 2B to 34B parameters and surpasses private models on some challenging MMB and MMMU evaluations.The reported comparisons include Gemini Pro, Qwen-VL-Plus, and GPT-4V across various settings.
2 Related Work
Related work traces multimodal modeling from large language models and vision-language integration to systems that connect LLMs with image generation. These approaches differ in how they align modalities and produce image outputs.
- Large Language Models: Large language models advanced language understanding and generation through Transformer-based architectures, autoregressive prediction, and instruction tuning.Instruction tuning was applied to open-source models including Alpaca and Vicuna.
- Vision Language Models: Vision-language models combine visual and linguistic models for cross-modal comprehension and reasoning.The related work cites models trained on diverse understanding and reasoning datasets, including CLIP as a foundational example.
- LLM as Generation Assistant: LLM-based image-output methods include retrieval-based interleaving and autoregressive image-token prediction.The cited autoregressive approaches decode images directly from large image-text datasets, requiring substantial training resources.
3 Mini-Gemini
Mini-Gemini uses dual vision encoders and patch-level information mining to add high-resolution visual detail while controlling visual-token processing. The design also supports token extension for richer visual representations.
- Framework: Mini-Gemini uses low-resolution embeddings and high-resolution candidates, with an LLM connecting text and images for comprehension and generation.The framework is designed as an any-to-any multimodal system.
- Dual Vision Encoders: The low-resolution image flow preserves long-range relations among N visual patches, while a CNN-based encoder processes high-resolution images efficiently.The high-resolution encoder produces a feature map by combining convolutional features at an upsampled scale.
- Patch Info Mining: Patch info mining uses low-resolution visual embeddings as queries and corresponding high-resolution regions as keys and values.Each low-resolution patch retrieves cues from a corresponding high-resolution sub-region containing M^2 pixel-wise features.
- Patch Info Mining: The mining operation synthesizes enhanced visual tokens while keeping their final count unchanged for efficient LLM processing.The design extracts high-resolution details without expanding the token count, balancing detail richness and computational feasibility.
- Visual Token Extension: Visual token extension can increase the token count to 5N by combining the original image with a 2× upscaled image.The CNN-based high-resolution encoder is designed to handle the augmented visual-token count.
3.3 Text and Image Generation
Mini-Gemini concatenates mined visual tokens with text tokens for autoregressive multimodal generation and supports text-only and text-image input-output workflows. It uses curated alignment, instruction, and generation data to extend these capabilities.
- Any-to-any Inference: Mini-Gemini supports text-only and text-image generation for both input and output, enabling any-to-any inference.Image generation is anchored on the model’s image-text understanding and reasoning capabilities.
- Text-image Instructions: Projector pretraining uses 558K LLaVA-filtered CC3M image-caption pairs and 695K ALLaVA GPT-4V captions, totaling about 1.2M captions.The data is collected for cross-modality alignment before instruction finetuning.
- Generation-related Instructions: A 13K GPT-4 Turbo instruction-following dataset supports image generation through re-captioning and in-context prompt-generation tasks.The tasks use descriptive captions and real-world conversation contexts to generate prompts for image production.
4 Experiments
Mini-Gemini is evaluated through zero-shot benchmarks, component ablations, and qualitative tests of visual understanding and image generation. Across these evaluations, high-resolution visual refinement, patch info mining, high-quality data, and scalable model configurations improve performance and broaden capabilities.
- Main Results: Mini-Gemini consistently outperforms existing models across normal-resolution settings and, with Hermes-2-Yi-34B, surpasses some private models on challenging benchmarks.The reported comparisons include MobileVLM, InstructBLIP, LLaVA-1.5, Qwen-VL-Plus, and Gemini Pro.
- Main Results: 74.1% TextVQA performance is achieved with Hermes-2-Yi-34B at high resolution while keeping the effective LLM visual-token count fixed.The high-resolution configuration uses 672 LR and 1536 HR input sizes; Mini-Gemini is also reported to surpass Gemini Pro and GPT-4V on MMB.
- Component-wise Analysis: 4.0% TextVQA and 18.1 MME gains result from patch info mining at LR 224 and HR 512, while further resolution increases produce additional improvements.At HR 768, TextVQA improves 5.7% over the baseline; with LR 336, gains remain consistent across TextVQA, MME, and MM-Vet.
- Component-wise Analysis: ConvNeXt-L provides the best balance of candidate quality and computational efficiency, whereas the larger ConvNeXt-XXL encoder causes convergence in performance.The encoder comparison finds ConvNeXt-L especially strong on MME and MM-Vet.
- Component-wise Analysis: High-quality captions, OCR-specific data, and generation-related instructions improve visual alignment, OCR reasoning, and MM-Vet performance.Generation-related data contributes a reported 3.3% gain on MM-Vet, while removing TextCaps data decreases TextVQA performance.
- Qualitative Results: Qualitative evaluations show visual reasoning over plots, indoor scenes, memes, charts, and intelligence tests, alongside instruction-aligned image generation.Figure 6 compares Mini-Gemini with ChatIllusion and AnyGPT and presents reasoning-generation cases with text generation.
5 Conclusion and Discussion
Mini-Gemini combines framework design, high-quality data, and expanded functionality to improve multimodal VLMs. The authors report strong benchmark performance while identifying unresolved challenges in visual comprehension and reasoning-based generation.
- Mini-Gemini uses patch info mining, enriched data, and any-to-any functionality to support image understanding, reasoning, and generation.The framework extracts detailed visual cues from high-resolution candidates and supports reasoning-based generation.
- Extensive zero-shot experiments show superiority over previous leading approaches and some private models.
- Counting and complex visual reasoning remain unsatisfactory, partly because corresponding pretraining data are lacking.
- Reasoning-based generation currently uses text to connect the VLM and diffusion model because embedding-based approaches showed no apparent gain.
A Data Collection Details
The data collection process targets difficult image-text understanding and generation cases. It incorporates OCR references for text-rich visuals and constructs a 13K instruction-following dataset for image generation.
- OCR-related data collection adds OCR tokens to support questions about documents, charts, and scientific diagrams.These text-rich figures are described as more challenging than natural images with detailed captions.
- 13K generation examples remain after filtering an initial corpus of 10K GPT4V captions and 6K English-only LLM SFT data.
- The generation dataset uses two tasks: simple instruction re-captioning and in-context prompt generation.The tasks are constructed with GPT-4 Turbo using image captions and real-world conversation contexts.
B Extended Showcases
Extended examples illustrate Mini-Gemini’s image understanding, reasoning, OCR handling, and diverse image-generation workflows. The showcases include prompt construction, visual question answering, mathematical reasoning, and interactive generation.
- Mini-Gemini answers detail-oriented questions and addresses OCR-related and scientific problems in qualitative image-understanding examples.
- The image-generation showcases cover direct text-to-image generation, multi-round conversation, reasoning-based generation, storytelling, and in-context generation.
- Figure 4 presents simple instruction re-captioning and in-context prompt generation as two pure-text data types for image generation.
- The data-generation prompt instructs GPT-4 to interpret descriptions, formulate queries, and produce diffusion-suitable captions.
- The generation-data illustration reports an approximate GPT-4 API cost of $80 for obtaining all 13K data points.
- The mathematical reasoning example identifies a student’s addition of Monday and Tuesday practice times while noting an omitted comparison between the days.