PaperlayerView exact revision on arXiv

Source-linked AI summary

Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, Jiaya Jia

arXiv:2403.18814v1cs.CVcs.AIcs.CL

TL;DR

Vision-language models still trail private systems such as GPT-4 and Gemini, while higher-resolution inputs can raise computational costs. Mini-Gemini combines dual-encoder visual processing, curated data, and any-to-any comprehension and generation, achieving leading zero-shot benchmark results and surpassing private models on selected challenging benchmarks.

  • Problem

    Academic vision-language models remain behind systems such as GPT-4 and Gemini, while increasing image resolution can raise computational costs and existing data quality and application scope remain inadequate.

    Source: §1 · p. 2

  • Method

    Mini-Gemini combines dual vision encoders and patch-level mining of high-resolution candidates with high-quality data and image-text comprehension and generation.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Mini-Gemini achieves leading results across several zero-shot benchmarks and surpasses prior leading and private models on selected challenging benchmarks.

    Source: §4.2 · p. 7, §5 · p. 11

  • Takeaways & Limitations

    Mini-Gemini's supported functional scope spans image understanding, reasoning, and generation in an any-to-any workflow.

    Source: §5 · p. 11

    The authors report that counting and complex visual reasoning remain unsatisfactory, and that text bridges the VLM and diffusion model because embedding-based approaches showed no apparent gain.

    Source: §5 · p. 11

Abstract

from arXiv · show

In this work, we introduce Mini-Gemini, a simple and effective framework enhancing multi-modality Vision Language Models (VLMs). Despite the advancements in VLMs facilitating basic visual dialog and reasoning, a performance gap persists compared to advanced models like GPT-4 and Gemini. We try to narrow the gap by mining the potential of VLMs for better performance and any-to-any workflow from three aspects, i.e., high-resolution visual tokens, high-quality data, and VLM-guided generation. To enhance visual tokens, we propose to utilize an additional visual encoder for high-resolution refinement without increasing the visual token count. We further construct a high-quality dataset that promotes precise image comprehension and reasoning-based generation, expanding the operational scope of current VLMs. In general, Mini-Gemini further mines the potential of VLMs and empowers current frameworks with image understanding, reasoning, and generation simultaneously. Mini-Gemini supports a series of dense and MoE Large Language Models (LLMs) from 2B to 34B. It is demonstrated to achieve leading performance in several zero-shot benchmarks and even surpasses the developed private models. Code and models are available at https://github.com/dvlab-research/MiniGemini.

1 Introduction

Mini-Gemini targets the gap between academic VLMs and models such as GPT-4 [4] and Gemini [5] through efficient high-resolution visual tokens, higher-quality data, and expanded image-text capabilities. The framework combines image understanding, reasoning, and generation while keeping visual token counts unchanged.

1 Introduction

  • The introduction identifies a gap between academic VLMs and models such as GPT-4 [4] and Gemini [5], while higher-resolution visual tokens increase computational demands and costs.It also describes data quality, model capabilities, and application scope as inadequate for accelerated development.Source: §1 · p. 2
  • Mini-Gemini uses a dual-encoder attention pipeline: low-resolution visual embeddings generate queries, while a high-resolution encoder supplies candidate keys and values without increasing the LLM’s visual token count.A ConvNet efficiently generates higher-resolution candidates to enhance visual detail.Source: §1 · p. 2
  • Mini-Gemini is designed for an any-to-any workflow that accepts and produces both images and text.Its approach brings visual understanding, reasoning, and generation together within manageable resource constraints.Source: §1 · p. 2
  • The framework combines public data including high-quality responses [14] [15], task-oriented instructions [16] [17] [18] [19], and generation-related data [20] to improve data quality and extend model capabilities.It also integrates the VLM with generative models [22] for concurrent image and text generation, using LLM-generated text to guide image generation.Source: §1 · p. 2
  • Mini-Gemini reports leading performance across settings and says it surpasses Gemini Pro [5], Qwen-VL-Plus [23], and GPT 4V [4] on the complex MMB [24] and MMU [25] datasets.The framework supports LLMs ranging from 2B to 34B parameters.Source: §1 · p. 2

2 Related Work

This related-work section situates Mini-Gemini among evolving LLMs and VLMs, then distinguishes its text-data-driven image-generation approach from prior retrieval-, autoregressive-, and diffusion-alignment methods.

Large Language Models

  • Instruction-tuned LLMs extend from foundational Transformer [26] and GPT [28] work to models including ChatGPT, GPT-4 [4], LLaMA [3], and Mixtral [29].The authors benchmark pretrained LLMs [3] [29] and build multimodal frameworks on them; Alpaca and Vicuna iterate on LLaMA [3] with custom instruction sets.Source: §2 · pp. 2–3

Vision Language Models

  • Prior VLMs combine vision and language for cross-modal comprehension and reasoning, while Mini-Gemini explicitly targets both comprehension and generation.Prior approaches include Flamingo [41] and BLIP-2 [6] for cross-modal alignment, BLIP-2-based instructional data [42] [8], and LLaVA [7] [43] with a linear projector and tailored instruction data.Source: §2 · p. 3

LLM as Generation Assistant

  • Mini-Gemini uses 13K pure-text data to activate high-quality re-captioning for image generation without undermining VLM text-generation performance.InternLM-XComposer [44] [45] retrieves images; EMU [46] [47] and SEED [48] [49] decode them autoregressively with substantial training-resource and latency costs, while latent-diffusion alignment [50] [51] [52] requires embedding design and optimization and can compromise VLM text generation.Source: §2 · p. 3

3 Mini-Gemini

Mini-Gemini combines low- and high-resolution visual encoding with patch-level feature mining to enrich visual tokens efficiently, then uses an LLM for image comprehension and generation. It supports flexible visual-token extension, any-to-any inference, and training data for image understanding and generation.

3.1 Dual Vision Encoders

  • The framework pairs a CLIP-pretrained ViT low-resolution flow with a CNN-based high-resolution flow, using ConvNeXt as an example HR encoder.The LR flow preserves long-range relations among visual patches, while the HR encoder processes images adaptively.Source: §3.1 · p. 4

3.2 Patch Info Mining

  • Patch info mining uses low-resolution patches as queries to retrieve and refine cues from corresponding high-resolution regions without increasing the final visual-token count.Each query is limited to its corresponding high-resolution sub-region, balancing richer detail with computational efficiency.Source: §3.2 · p. 4
  • Patch info mining can extend visual tokens to 5N by processing the original image and its 2× upscaled counterpart, which the CNN-based HR encoder can accommodate.The method batches five low-resolution inputs and adjusts the high-resolution sub-region to the expanded embedding.Source: §3.2 · p. 4

3.3 Text and Image Generation

  • Mini-Gemini concatenates mined visual tokens with text tokens for LLM generation and supports text-only and text-image inputs and outputs in any-to-any inference.For image generation, it translates user instructions into high-quality language prompts for latent diffusion models.Source: §3.3 · p. 5
  • Training data includes about 1.2M image captions for projector pretraining, about 1.5M image-comprehension instruction conversations, and 13K image-related generation pairs.The instruction data includes 28K OCR-related QA pairs; the generation pairs are elaborated in the generation-instruction procedure.Source: §3.3 · p. 5
  • The 13K generation-instruction dataset combines 8K simple instruction re-caption examples with 5K in-context prompt-generation instructions.GPT-4 Turbo creates target prompts using five sampled GigaSheet text-to-image prompts as in-context examples; Mini-Gemini uses the <GEN> trigger and SDXL to generate images from extracted captions.Source: §3.3 · p. 5

4 Experiments

Mini-Gemini outperforms comparable methods across zero-shot benchmarks, with high-resolution visual information and strategic data integration contributing to performance gains. Component analyses and qualitative examples also demonstrate its visual reasoning and image-generation capabilities.

4.1 Experimental Setup

  • Mini-Gemini’s setup freezes both vision encoders, trains projectors throughout and the LLM during instruction tuning, using one epoch of AdamW with cosine scheduling.Training uses 1.2M caption pairs and 1.5M instruction-tuning conversations; evaluation covers TextVQA, MMB, MME, MM-Vet, MMMU, and MathVista.Source: §4.1 · p. 6

4.2 Main Results

  • With the same LLM, Mini-Gemini surpasses LLaVA-1.5 across benchmarks, while Hermes-2-Yi-34B exceeds private models on some challenging benchmarks.In the efficient model category, the Gemma-2B configuration also outperforms MobileVLM and InstructBLIP with Vicuna-7B and 13B.Source: §4.2 · pp. 6–7, Table 1 · p. 6
  • 74.1% on TextVQA with Hermes-2-Yi-34B at high resolution closely matches Gemini Pro; Mini-Gemini also surpasses Gemini Pro and GPT-4V on MMB.The HR encoder supplies high-resolution candidates while the LLM processes the same effective number of visual tokens as with the 672 LR input.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 Component-wise Analysis

  • Against the baseline, patch info mining at LR 224 and HR 512 adds 4.0% on TextVQA and 18.1 on MME; at HR 768, it adds 5.7% on TextVQA.At LR 336 with ConvNeXt-L, gains over baseline are 3.3% on TextVQA, 6.3 on MME, and 3.5% on MM-Vet.Source: §4.3 · p. 7, Table 2 · p. 7
  • ConvNeXt-L delivers peak results especially on MME and MM-Vet, while ConvNeXt-XXL converges; the study selects ConvNeXt-L for its performance-efficiency balance.ConvNeXt-B performs better than ConvNeXt-L on TextVQA and MM-Vet, but not consistently across the evaluated benchmarks.Source: Table 2 · p. 7, §4.3 · p. 8
  • 3.3% MM-Vet gains follow generation-instruction data; ALLaVA GPT4V responses push the baseline over 7% on TextVQA and 9% on MM-Vet.Removing TextCaps data decreases TextVQA performance; additional LAION-GPT-4V captions and OCR-specific data enhance OCR reasoning.Source: Table 3 · p. 7, §4.3 · p. 8
  • Increasing LR and HR input resolution improves all benchmarks, with over 3% uplift on TextVQA; the authors report reduced visual hallucinations.The analysis associates increased visual-token numbers with better capability and notes that patch info mining accommodates extended tokens.Source: Table 3 · p. 7, §4.3 · p. 8

4.4 Qualitative Results

  • Qualitative examples show visual understanding across charts, plotted curves, indoor scenes, memes, and problem-solving, alongside instruction-aligned image-text generation in single- or multi-round conversations.The generation examples use text-only training data and compare outputs with ChatIllusion and AnyGPT.Source: §4.4 · p. 8, Figure 5 · p. 9, Figure 6 · p. 10

5 Conclusion and Discussion

Mini-Gemini combines high-resolution patch information, high-quality data, and reasoning-based generation to expand VLM capabilities. It outperforms previous leading approaches and private models on several zero-shot benchmarks, while counting and complex visual reasoning remain unsatisfactory.

5 Conclusion and Discussion

  • Mini-Gemini surpasses previous leading approaches and private models on several zero-shot benchmarks.The authors hope it will serve as a strong benchmark for image understanding and VLM-guided generation.Source: §5 · p. 11
  • The framework combines high-resolution patch information, high-quality vision-language data, and reasoning-based generation in an any-to-any workflow.Patch info mining extracts detailed visual cues from high-resolution candidates; the dataset supports accurate alignment and instruction following.Source: §5 · p. 11
  • Counting and complex visual reasoning remain unsatisfactory, which the authors attribute in part to insufficient corresponding pretraining data.The authors identify these capabilities as areas for further exploration.Source: §5 · p. 11
  • For reasoning-based generation, Mini-Gemini uses text to connect the VLM and diffusion model because embedding-based approaches showed no apparent gain.The authors plan to seek a more advanced approach to visual understanding, reasoning, and generation.Source: §5 · p. 11

Appendix · A Data Collection Details

The appendix describes OCR-enriched image-text training data and a filtered generation-data collection, including their sources and prompt construction. OCR is used to enrich training conversations but not during testing.

A Data Collection Details

  • OCR tokens are appended to original conversations as “Reference OCR token:Text_1,...,Text_n” to provide textual representations of image content during training.PaddleOCR identifies and extracts text strings from each image.Source: A Data Collection Details · p. 11
  • The OCR detector generates enriched training data only and is not used during testing.The collection strategy follows TextVQA [67] by incorporating OCR tokens for model reference in training.Source: A Data Collection Details · p. 11
  • Text-rich documents [17], charts [18], and scientific diagrams [19] are described as more challenging to annotate and handle in complex question-answering than natural images.Source: A Data Collection Details · p. 11
  • 13K generation-data points remained after filtering a corpus of 10K GPT4V captions and 6K English-only LLM SFT data for format requirements.The data collection supports the generation tasks described in Section 3.3.Source: A Data Collection Details · p. 11
  • Generation queries use two types of in-context examples: five randomly selected high-quality Stable Diffusion T2I prompts and three instances from simple instructional templates.The examples are drawn through get_example_captions() and get_example_queries(), respectively.Source: A Data Collection Details · p. 11
  • The generation-data examples provide reference sources and query prompts for two generation tasks, illustrated in Figure 7.Source: A Data Collection Details · p. 11

B Extended Showcases

The extended showcases present Mini-Gemini examples of high-resolution visual understanding and image generation across varied tasks. They include detail-oriented, OCR-related, and scientific questions, as well as direct and interactive generation, reasoning, storytelling, and in-context generation.

B Extended Showcases

  • Figure 10’s generation examples include direct text-to-image, multi-round conversation, reasoning-based generation, storytelling, and in-context generation.The paper presents these as further examples of Mini-Gemini’s image-generation capabilities.Source: B Extended Showcases · p. 11, Figure 10 · p. 15
  • Figures 8 and 9 show Mini-Gemini answering detail-oriented questions and handling OCR-related and scientific problems, while Figure 8 also highlights a failed image-understanding case.The paper presents these examples as evidence of Mini-Gemini’s capability across varied environments.Source: B Extended Showcases · p. 11, Figure 8 · p. 13, Figure 9 · p. 14
  • On interactive image generation, Mini-Gemini is described as outperforming ChatIllusion [51] and LLMGA [50] in adaptability, detail capture, and coherence without further text-image alignment tuning.Figure 10 also shows plug-and-play generation, reasoning-based generation, and multi-round storytelling; generated images use SDXL unless otherwise specified.Source: Figure 10 · p. 15

Related papers

Loading 2403.18814v1…