Source-linked AI summary
MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe
Tianyu Yu, Zefan Wang, Chongyi Wang, Fuwei Huang, Wenshuo Ma, Zhihui He, Tianchi Cai, Weize Chen, Yuxiang Huang, Yuanqian Zhao, Bokai Xu, Junbo Cui, Yingjing Xu, Liqing Ruan, Luoyuan Zhang, Hanyu Liu, Jingkun Tang, Hongyuan Liu, Qining Guo, Wenhao Hu, Bingxiang He, Jie Zhou, Jie Cai, Ji Qi, Zonghao Guo, Chi Chen, Guoyang Zeng, Yuxuan Li, Ganqu Cui, Ning Ding, Xu Han, Yuan Yao, Zhiyuan Liu, Maosong Sun
TL;DR
MLLMs face escalating data-engineering, training, and inference costs that limit accessibility and scalability. MiniCPM-V 4.5 addresses these costs with unified visual compression, direct document/OCR learning, and hybrid reinforcement learning, achieving strong performance with an 8B-parameter model. It surpasses larger or proprietary comparison models in reported evaluations while emphasizing efficiency.
Problem
Rising data-engineering, training, and inference costs are a central efficiency bottleneck for making capable MLLMs more accessible and scalable.
Method
MiniCPM-V 4.5 combines a unified 3D-Resampler, direct document-image learning for knowledge and OCR, and hybrid post-training for short and long reasoning modes.
Results
MiniCPM-V 4.5 scores 77.0 on OpenCompass with 8B parameters and surpasses GPT-4o-latest and Qwen2.5-VL 72B for vision-language capabilities.
Takeaways & Limitations
The model demonstrates a promising path toward reducing MLLM efficiency bottlenecks while retaining strong multimodal performance and controllable reasoning.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) are undergoing rapid progress and represent the frontier of AI development. However, their training and inference efficiency have emerged as a core bottleneck in making MLLMs more accessible and scalable. To address the challenges, we present MiniCPM-V 4.5, an 8B parameter model designed for high efficiency and strong performance. We introduce three core improvements in model architecture, data strategy and training method: a unified 3D-Resampler model architecture for highly compact encoding over images and videos, a unified learning paradigm for document knowledge and text recognition without heavy data engineering, and a hybrid reinforcement learning strategy for proficiency in both short and long reasoning modes. Comprehensive experimental results in OpenCompass evaluation show that MiniCPM-V 4.5 surpasses widely used proprietary models such as GPT-4o-latest, and significantly larger open-source models such as Qwen2.5-VL 72B. Notably, the strong performance is achieved with remarkable efficiency. For example, on the widely adopted VideoMME benchmark, MiniCPM-V 4.5 achieves state-of-the-art performance among models under 30B size, using just 46.7\% GPU memory cost and 8.7\% inference time of Qwen2.5-VL 7B.
1 Introduction
MiniCPM-V 4.5 targets the rising data, training, and inference costs of capable MLLMs with architectural, data, and training improvements. It reports strong benchmark performance alongside compact visual encoding and controllable reasoning.
- Motivation: MLLM efficiency is constrained by growing data-engineering, training, and inference costs, especially from long visual token sequences.A 6-second, 2-fps 448×448 video requires 1,536 tokens in Qwen2.5-VL and 3,072 in InternVL3.
- Contributions: MiniCPM-V 4.5 introduces a unified 3D-Resampler, document/OCR learning paradigm, and hybrid post-training strategy.The three improvements address image/video encoding, document knowledge and text recognition, and short/long reasoning.
- Contributions: 128 visual tokens encode a 6-second, 2-fps, 448×448 video, reducing token cost by 12×-24× versus representative MLLMs.The 3D-Resampler jointly compresses spatial-temporal information for videos and provides unified image encoding.
- Results: OpenCompass results show MiniCPM-V 4.5 surpassing GPT-4o-latest and Qwen2.5-VL 72B while supporting efficient high-frame-rate and long-video understanding.The model is also described as supporting controllable hybrid reasoning and robust OCR and document parsing.
- Architecture: The architecture processes images and videos through a visual encoder, unified 3D-Resampler, and LLM decoder with short or long reasoning outputs.The resampler achieves up to 16× image compression and an additional 6× video compression.
2 Approach
The approach combines unified visual compression, progressive pre-training, and direct document-image learning. Dynamic corruption unifies OCR and document knowledge acquisition while avoiding fragile parsing pipelines.
- Unified 3D-Resampler: A unified 3D-Resampler jointly compresses spatial-temporal video information and serves as a shared image-video encoding architecture.It leverages temporal redundancy and can be adapted from the 2D-Resampler through lightweight SFT.
- Unified 3D-Resampler: 96× video-token compression represents six 448×448 frames with 64 tokens instead of the 1,536-3,072 tokens typical of many MLLMs.The compact representation supports perceiving more video frames without increasing LLM inference cost.
- Pre-training: Progressive pre-training successively trains the resampler, vision encoder, and full model while introducing increasingly complex data.The three stages move from image-caption alignment to perceptual learning and end-to-end multimodal training.
- Document Knowledge and OCR: Direct document-image learning avoids complex parsing pipelines, reduces parser noise, and combines knowledge and OCR objectives in the same training batch.The approach is described as producing a versatile model for document understanding.
- Document Knowledge and OCR: The document paradigm predicts original text from dynamically corrupted document images across varying corruption levels.Low corruption supports OCR, while heavier corruption requires contextual inference and integrated visual-context reasoning.
2.3 Supervised Fine-tuning
Supervised fine-tuning first establishes broad multimodal interaction, then adds long reasoning, temporal understanding, and specialized reasoning skills through targeted data and architecture upgrades.
- General SFT: SFT begins with diverse instruction-response data and includes 10% high-quality text-only data to preserve text performance and improve stability.This general SFT stage activates pretrained knowledge for multimodal interaction.
- Long-CoT and 3D-Resampler: The second SFT stage introduces Long-CoT warm-up instructions and upgrades the architecture from 2D- to 3D-Resampler with high-frame-rate and long-video data.These changes cultivate explicit step-by-step reasoning and temporal understanding.
- Specialized Data: STEM data is filtered for high visual dependency and answer consistency before training.The covered disciplines include physics, chemistry, biology, finance, and computer science.
- Specialized Data: Long-tail knowledge data synthesizes multimodal instruction-following examples from Wikipedia entity pages and retains samples with high visual dependency.The data targets less common topics where models often fail.
- Long-CoT Data: Long-CoT responses are drawn from challenging prompts and validated for correctness, factuality, and non-repetition before diversity augmentation.The pipeline filters prompts on which early-stage models struggle and applies multistage response validation.
2.4 Reinforcement Learning
The reinforcement-learning recipe combines quality-controlled rewards with jointly optimized short and long reasoning modes. It also applies visual-feedback alignment to improve factual grounding across image and video inputs.
- RL Strategy: The RL stage combines rule-verified, probability-based, calibrated preference, and RLAIF-V rewards to improve reasoning, controllability, and trustworthiness.Rule-based rewards target straightforward cases, while probability-based rewards address complex answers.
- Reward Quality Control: Rule-based verification achieves 98% reward accuracy for short straightforward answers, while RLPR probability-based rewards handle complex natural-language responses.The validation method is selected dynamically according to answer complexity.
- Hybrid Reasoning: Hybrid RL jointly optimizes prompt-controlled short reasoning and explicit step-by-step long reasoning, with rollouts alternating between modes.The design is intended to preserve quick responses while retaining complex reasoning capabilities.
- Reward Shaping: The preference reward model scores only the final answer in long reasoning mode to avoid instability from evaluating out-of-distribution reasoning chains.This selective application supplies a dense preference-aligned signal without penalizing explicit thinking steps.
- Visual Grounding: RLAIF-V decomposes responses into atomic claims, constructs preference pairs from claim verification, and trains on both image and video data with DPO.The procedure targets factual accuracy and visual grounding.
- Takeaways: Combining simple-response rule rewards with complex-response probability rewards enables reliable rewards across diverse tasks and supports cross-mode generalization.The stated takeaway links reward specialization with transfer between long and short reasoning modes.
3 Experiments
MiniCPM-V 4.5 is evaluated against strong open-source, comparable-size, and proprietary models across diverse multimodal benchmarks. Results show broad capability, reduced inference cost, and benefits from hybrid reinforcement learning, probability-based rewards, unified document learning, and the 3D-Resampler.
- Baselines and Benchmarks: The evaluation spans STEM, document and OCR, hallucination, multi-image, video, and comprehensive multimodal understanding benchmarks.Baselines include Qwen2.5-VL 72B, InternVL3 8B, GLM-4.1V 9B, and GPT-4o.
- Main Results: 77.0 OpenCompass average: MiniCPM-V 4.5 surpasses GPT-4o-latest and Qwen2.5-VL 72B with only 8B parameters.The average covers eight popular benchmarks.
- Main Results: MiniCPM-V 4.5 achieves leading OCRBench performance and state-of-the-art general-MLLM PDF parsing on OmniDocBench.The model also shows strong high-frame-rate and competitive long-video understanding performance.
- Main Results: 42.9% of GLM-4.1V's time: MiniCPM-V 4.5 completes OpenCompass evaluation with the highest average score among models under 30B.Efficiency evaluation uses a standard configuration of 8 A100 GPUs and compares inference time and GPU memory consumption.
- Ablations: 70.5% of long-reasoning-only training token cost: hybrid reinforcement learning achieves better OpenCompass performance using half as many long-reasoning samples.The strategy mixes short- and long-reasoning samples during training.
- Ablations: The ablations support probability-based rewards, unified document learning, and 3D-Resampler encoding as effective design choices.Probability-based rewards complement rule verification; unified learning improves knowledge-intensive and text-recognition tasks; the 3D-Resampler uses one-third as many visual tokens per frame as the 2D baseline.
4 Conclusion
MiniCPM-V 4.5 targets MLLM efficiency through coordinated architectural, data, and training improvements. Its unified 3D-Resampler, document learning paradigm, and hybrid post-training strategy support efficient multimodal understanding and reasoning.
- 4 Conclusion: The unified 3D-Resampler jointly compresses spatial-temporal information for efficient high-frame-rate and long-video understanding.The model is designed for strong performance with superior encoding efficiency.
- 4 Conclusion: Unified learning lets the model learn document knowledge and text recognition directly from document images, bypassing fragile parsers.This significantly reduces data-engineering complexity.
- 4 Conclusion: Hybrid post-training improves training and inference efficiency while facilitating generalization between short and long reasoning modes.The strategy supports controllable reasoning across both modes.
- 4 Conclusion: Overall, MiniCPM-V 4.5 demonstrates a promising path toward addressing efficiency bottlenecks in MLLM development.
A Implementation Details
The implementation uses staged learning-rate schedules for pre-training, supervised fine-tuning, and the Long-CoT/3D-Resampler stage, followed by configured reinforcement-learning phases.
- A Implementation Details: Pre-training uses a WSD schedule, with the learning rate fixed at 5 × 10−5 during the stable phase before decaying to 1 × 10−5.SFT instead applies cosine decay from 1 × 10−5 to 1 × 10−6.
- A Implementation Details: The Long-CoT and 3D-Resampler stage continues from the SFT checkpoint, warming up to 5 × 10−6 and decaying to 1 × 10−6.
- A Implementation Details: RL uses GRPO with 128 prompts, 8 responses per prompt, an 8192-token maximum response length, and 50% long-reasoning prompts.Rollouts use temperature 1.0 and a fixed learning rate of 1×10−6, without entropy loss or KL penalty.
- A Implementation Details: RLAIF-V uses a global batch size of 256, learning rate 1 × 10−6, β = 0.1, and 400 steps.
B.1 Comprehensive Instruction Following
The evaluation includes cases of comprehensive instruction following and creative writing in Chinese. These examples cover both general reasoning and Chinese-language generation.
- B.1 Comprehensive Instruction Following: Figure 4 presents a case of comprehensive real-world reasoning.
- B.1 Comprehensive Instruction Following: Figure 5 presents a case of comprehensive real-world reasoning in Chinese.
- B.1 Comprehensive Instruction Following: Figure 6 presents a case of creative writing in Chinese.
B.2 World Knowledge
The evaluation includes world-knowledge understanding cases in both general and Chinese-language settings.
- B.2 World Knowledge: Figure 7 presents a case of world knowledge understanding.
- B.2 World Knowledge: Figure 8 presents a case of world knowledge understanding in Chinese.
B.3 OCR
This section presents OCR-related examples spanning handwritten text recognition, table content extraction, chemistry problem solving, and multi-image statistical problem solving.
- Handwritten text recognition is illustrated in both Chinese and non-Chinese cases.
- A table content extraction example demonstrates document-structure understanding.
- A chemistry problem-solving case is shown in Chinese.
- A multi-image statistical problem-solving case extends the examples beyond single-image OCR.