PaperlayerView exact revision on arXiv

Source-linked AI summary

MiniCPM-V: A GPT-4V Level MLLM on Your Phone

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun

arXiv:2408.01800v1cs.CV

TL;DR

MLLMs’ large parameter and computation requirements limit practical use on mobile, offline, energy-sensitive, and privacy-protective devices. This paper introduces MiniCPM-V, combining efficient end-side architecture, training, alignment, and deployment techniques. MiniCPM-Llama3-V 2.5 reports GPT-4V-level performance with fewer parameters, strong OCR and high-resolution perception, lower hallucination rates, multilingual support, and mobile usability.

  • Problem

    MLLMs require massive parameters and heavy computation, limiting deployment beyond high-performing cloud servers and restricting practical end-side applications.

  • Method

    The paper introduces MiniCPM-V, a series of efficient end-side MLLMs integrating architecture, pretraining, alignment, and deployment techniques.

  • Results

    MiniCPM-Llama3-V 2.5 achieves GPT-4V-level performance with significantly fewer parameters, strong OCR and 1.8M-pixel perception, lower hallucination rates than GPT-4V-1106, and acceptable mobile-phone usability.

  • Takeaways & Limitations

    The paper presents MiniCPM-V as evidence that powerful, efficient, safe, and trustworthy MLLMs are becoming increasingly possible on end-side devices.

  • Takeaways & Limitations

    Current models still have room to improve multimodal understanding and inference efficiency, and remain limited to image modality rather than video or audio.

Abstract

from arXiv · show

The recent surge of Multimodal Large Language Models (MLLMs) has fundamentally reshaped the landscape of AI research and industry, shedding light on a promising path toward the next AI milestone. However, significant challenges remain preventing MLLMs from being practical in real-world applications. The most notable challenge comes from the huge cost of running an MLLM with a massive number of parameters and extensive computation. As a result, most MLLMs need to be deployed on high-performing cloud servers, which greatly limits their application scopes such as mobile, offline, energy-sensitive, and privacy-protective scenarios. In this work, we present MiniCPM-V, a series of efficient MLLMs deployable on end-side devices. By integrating the latest MLLM techniques in architecture, pretraining and alignment, the latest MiniCPM-Llama3-V 2.5 has several notable features: (1) Strong performance, outperforming GPT-4V-1106, Gemini Pro and Claude 3 on OpenCompass, a comprehensive evaluation over 11 popular benchmarks, (2) strong OCR capability and 1.8M pixel high-resolution image perception at any aspect ratio, (3) trustworthy behavior with low hallucination rates, (4) multilingual support for 30+ languages, and (5) efficient deployment on mobile phones. More importantly, MiniCPM-V can be viewed as a representative example of a promising trend: The model sizes for achieving usable (e.g., GPT-4V) level performance are rapidly decreasing, along with the fast growth of end-side computation capacity. This jointly shows that GPT-4V level MLLMs deployed on end devices are becoming increasingly possible, unlocking a wider spectrum of real-world AI applications in the near future.

1 Introduction

MiniCPM-V targets the practical deployment gap caused by MLLMs’ parameter and computation demands, pursuing a balance between performance and efficiency on end-side devices. MiniCPM-Llama3-V 2.5 combines strong benchmark, OCR, trustworthiness, multilingual, and mobile-deployment results with a broader trend toward smaller GPT-4V-level models.

  • Cloud-oriented MLLMs’ massive parameter counts and computational burdens restrict deployment in mobile, offline, energy-sensitive, and privacy-protective scenarios.
  • MiniCPM-V presents efficient end-side MLLMs designed to balance performance and efficiency under constrained parameter and inference-computation budgets.The series is introduced as a practical approach for mobile phones and other end-side devices.
  • The paper identifies a trend in which models reaching GPT-4V-level performance are rapidly shrinking while end-side computation capacity increases.The authors describe the convergence as bringing usable end-side MLLMs within reach.
  • MiniCPM-Llama3-V 2.5 outperforms GPT-4V-1106, Gemini Pro, and Claude 3 on OpenCompass, an evaluation spanning 11 popular benchmarks.
  • The work introduces and open-sources MiniCPM-V, investigates techniques for performance-efficiency balance, and empirically illustrates the proposed MLLM scaling trend.

2 Related Works

Related work traces MLLMs from visual-language systems built on pretrained LLMs toward smaller end-side models. The paper positions MiniCPM-V within this efficiency-oriented movement and illustrates its capabilities through qualitative mobile-use examples.

  • Early MLLMs connected pretrained visual encoders with large language models, while later open-source systems commonly built on Llama- or Vicuna-family LLMs.
  • Recent MLLM improvements include high-resolution input, additional training data, and revised data ratios.
  • Qualitative examples cover reading image text, converting tables to Markdown, complex reasoning, and multilingual interaction on mobile phones.
  • Smaller LLMs such as Phi, Gemma, MobileLLM, and MiniCPM make end-side deployment on personal computers and mobile phones more feasible.The related-work passage associates their moderate size with end-side applicability.

3 Model Architecture

MiniCPM-V uses a visual encoder, token-compression layer, and LLM, with adaptive visual encoding for high-resolution images of varied aspect ratios. Its design limits visual-token and resolution costs to support end-side efficiency while retaining image detail.

  • The architecture consists of a visual encoder, a shared compression layer, and an LLM that receives compressed visual tokens alongside text.The visual encoder uses SigLIP SoViT-400m/14, while compression uses a one-layer cross-attention perceiver resampler.
  • Adaptive visual encoding addresses the trade-off between preserving aspect ratio and high-resolution detail while keeping visual-token counts affordable on end-side devices.
  • Adaptive Visual Encoding: Images are partitioned into slices whose number is estimated from the input and ViT pretraining resolutions, then arranged through candidate row-column combinations.
  • Adaptive Visual Encoding: The selected partition maximizes a score over feasible slice arrangements, with neighboring candidate sets added when the ideal slice count is prime.
  • Adaptive Visual Encoding: N < 10 supports up to 1.8M pixels, such as 1344 × 1344, while limiting further resolution because its benefit is marginal relative to overhead.
  • Adaptive Visual Encoding: Each slice is resized to match the ViT’s pretraining area before encoding, helping keep encoded-patch counts aligned with pretraining.
  • Adaptive Visual Encoding: Slice representations are compressed from 1,024 visual tokens to 96 tokens per slice in MiniCPM-Llama3-V 2.5, using cross-attention with positioned queries.
  • Adaptive Visual Encoding: Special slice and row-separator tokens preserve each slice’s position relative to the whole image.

4 Training

MiniCPM-V training combines three phases—pre-training, supervised fine-tuning, and RLAIF-V—with staged visual-module training, curated data, multilingual transfer, and preference optimization for reduced hallucination.

  • Three phases structure training: pre-training, supervised fine-tuning, and RLAIF-V alignment.
  • Pre-training: Pre-training aligns the visual encoder and compression layer with the LLM while learning foundational multimodal knowledge from large-scale image-text pairs.
  • Pre-training: The visual modules progress from compression-layer warm-up at 224×224, through 448×448 encoder training, to adaptive encoding for high-resolution images with any aspect ratio.
  • Pre-training: Caption rewriting, fixed-length data packing, and multilingual transfer improve data quality, training efficiency, and cross-language multimodal capability.
  • Supervised fine-tuning: Supervised fine-tuning uses high-quality visual question-answering data with all parameters unlocked, combining basic recognition examples and detailed instruction-following examples.
  • RLAIF-V: RLAIF-V generates multiple policy-model responses, scores atomic-claim correctness using divide-and-conquer feedback, and applies DPO to preference pairs.

5 End-side Deployment

End-side deployment addresses constrained memory and compute through quantization, framework selection, and device-specific optimizations. These techniques enable operation across phones and computers, although LLM prefilling remains the main bottleneck.

  • Challenges: End-side devices constrain MLLM deployment through limited memory, heat dissipation, size, power consumption, and slower CPU/GPU computation than servers.
  • Basic Practice: Quantization compresses weights using scaling and discretization; Q4_K_M 4-bit quantization reduces MiniCPM-Llama3-V 2.5 memory from 16~17G to around 5G.
  • Basic Practice: On Xiaomi 14 Pro with quantization and llama.cpp, text encoding latency is 64.2s and decoding speed is 1.3 tokens/s, remaining far from acceptable.
  • Advanced Practice: Memory, compilation, configuration, and NPU optimizations target image processing, device-specific execution, CPU allocation, and visual encoding.
  • Advanced Practice: Compilation optimization reduces encoding latency from 50.5s to 17.0s and increases decoding throughput from 1.3 tokens/s to 3.2 tokens/s.
  • Results: Across Xiaomi 14 Pro, vivo X00 Pro, and Macbook Pro, optimized deployment delivers acceptable latency and throughput, with Xiaomi’s NPU achieving Mac M1-like encoding speed.
  • Results: LLM prefilling remains the primary computation bottleneck, motivating fewer visual tokens and better GPU/NPU acceleration for LLM encoding.

6 Experiments

Experiments show that MiniCPM-Llama3-V 2.5 combines strong general, OCR, multilingual, and trustworthiness performance with lower computation requirements. Ablations attribute gains to RLAIF-V and multilingual generalization, while case studies demonstrate high-resolution and practical multimodal capabilities.

  • General multimodal benchmarks: MiniCPM-Llama3-V 2.5 surpasses Idefics2-8B by 7.9 points on OpenCompass while also outperforming substantially larger open-source models.OpenCompass aggregates 11 popular multimodal benchmarks.
  • General multimodal benchmarks: MiniCPM-Llama3-V 2.5 achieves better OpenCompass performance than GPT-4V-1106 and Gemini Pro with significantly fewer parameters, while recording lower hallucination rates than GPT-4V-1106 on Object HalBench.These results support the model’s reported balance between performance and efficiency.
  • OCR and multimodal capabilities: MiniCPM-Llama3-V 2.5 outperforms open-source models from 1.7B to 34B on OCRBench, TextVQA, and DocVQA, while performing comparably to GPT-4V-1106 and Gemini Pro.The model also transcribes screenshots and converts mixed-language tables into Markdown.
  • Multilingual capability: MiniCPM-Llama3-V 2.5 extends multimodal capability to over 30 languages and outperforms Yi-VL 34B and Phi-3-vision-128k-instruct on multilingual LLaVA Bench.Case studies show multimodal conversations in German, French, Japanese, Korean, and Spanish.
  • Efficiency and case studies: MiniCPM-Llama3-V 2.5 uses 96–960 visual tokens versus 1728–2880 for LLaVA-NeXT-Llama-3-8B, reducing inference computation relevant to end-side deployment.The paper connects this difference to inference speed, first-token latency, memory usage, and power consumption.
  • Ablation studies: RLAIF-V reduces response-level and mention-level hallucination rates while improving OpenCompass performance by 0.6 points across 11 benchmarks.The ablation reports no sacrifice in general capabilities.
  • Ablation studies: Less than 0.5% multilingual SFT data produces over 25 point improvement in all evaluated languages, although gains vary across languages.The authors attribute the uneven gains to factors such as the base LLM’s ability in each language.
  • Efficiency and case studies: The model processes high-resolution images with an extreme 10:1 aspect ratio and can interpret images within images.The case study reports accurate recognition of fine-grained article contents and description of a central image.

7 Conclusion

The paper presents MiniCPM-V as an efficient end-side MLLM series that reaches GPT-4V-level performance with substantially fewer parameters and an acceptable mobile-phone experience. It also acknowledges remaining limitations in capability depth, modality breadth, inference latency, battery dependence, and deployment infrastructure.

  • Conclusion: MiniCPM-V achieves GPT-4V-level performance with significantly fewer parameters through adaptive visual encoding, multilingual generalization, and RLAIF-V.End-side optimization techniques are used to support an acceptable user experience on mobile phones.
  • Limitations: Current models still have room to improve multimodal understanding and inference efficiency and to expand beyond images into video and audio.The paper identifies these as capability-depth and capability-width limitations.
  • Limitations: End-side deployment remains constrained by insufficient inference speed and latency, battery capacity, and hardware or framework optimization targeted mainly at CNNs and LSTMs.The authors identify tailored MLLM deployment efforts as an area with substantial improvement potential.
  • Future work: The authors anticipate progress in both model capabilities and smartphone chips and deployment frameworks as a path toward more satisfying end-side user experiences.Future work addresses capability depth, capability width, and device-side infrastructure together.
Loading 2408.01800v1…