Source-linked AI summary
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun
TL;DR
Large language models are expensive to train and difficult to deploy efficiently, motivating investigation of capable Small Language Models. MiniCPM combines model wind tunnel experiments with the WSD scheduler to scale training across model and data dimensions, while its small variants and family models show strong benchmark performance. The paper supports SLMs as a resource-efficient basis for studying and developing larger models, but does not yet validate its scaling law by training an LLM.
Problem
Training large language models is financially burdensome and operationally inefficient, while deployment on personal computers or smartphones is often infeasible.
Method
MiniCPM uses model wind tunnel experiments for model scaling and a Warmup-Stable-Decay scheduler for continuous, data-scalable training.
Results
MiniCPM’s foundation models surpass Mistral-7B and Llama-13B, while its DPO, 128K, and MoE variants match or exceed larger comparison models on reported benchmarks.
Takeaways & Limitations
MiniCPM demonstrates the potential of Small Language Models for practical applications and for informing scalable development of larger language models.
Takeaways & Limitations
The paper does not train an LLM to validate its scaling law, and WSD application to LLMs remains insufficiently explored.
Abstract
from arXiv · showhide
The burgeoning interest in developing Large Language Models (LLMs) with up to trillion parameters has been met with concerns regarding resource efficiency and practical expense, particularly given the immense cost of experimentation. This scenario underscores the importance of exploring the potential of Small Language Models (SLMs) as a resource-efficient alternative. In this context, we introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter variants, not only excel in their respective categories but also demonstrate capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach exhibits scalability in both model and data dimensions for future LLM research. Regarding model scaling, we employ extensive model wind tunnel experiments for stable and optimal scaling. For data scaling, we introduce a Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to continuous training and domain adaptation. We present an in-depth analysis of the intriguing training dynamics that occurred in the WSD LRS. With WSD LRS, we are now able to efficiently study data-model scaling law without extensive retraining experiments on both axes of model and data, from which we derive the much higher compute optimal data-model ratio than Chinchilla Optimal. Additionally, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE and MiniCPM-128K, whose excellent performance further cementing MiniCPM's foundation in diverse SLM applications. MiniCPM models are available publicly at https://github.com/OpenBMB/MiniCPM .
1 Introduction
MiniCPM addresses the cost and deployment barriers of trillion-parameter-scale language models by exploring capable Small Language Models. Its 1.2B and 2.4B variants demonstrate strong comparative performance, scalable training strategies, and an expanded family of specialized models.
- Training and deploying very large language models is financially burdensome, operationally inefficient, and often infeasible on personal computers or smartphones.
- Small Language Models offer efficient practical deployment and may guide the development of larger models when trained with scalable strategies.
- MiniCPM introduces 1.2B and 2.4B non-embedding-parameter models that rank highly in their respective scale categories and show capabilities comparable to 7B–13B models.
- Model wind tunnel experiments and the WSD learning-rate scheduler support scalable model and data training, including continuous training and checkpoint reuse.
- The MiniCPM family extends to DPO, long-context, and MoE variants, broadening its applications as a Small Language Model family.
- MiniCPM presents Small Language Models as evidence of latent capability and as a basis for more scientific and sustainable LLM scaling.
2 Related Work
Related work frames Small Language Models as deployable alternatives to larger models and reviews strategies for improving their capabilities. It also situates MiniCPM within broader efforts to make language-model scaling more predictable and stable.
- Small Language Models: Small Language Models are generally smaller than LLMs, typically below 7 billion parameters, and can run on personal computers or smartphones without GPUs.
- Small Language Models: Prior SLM improvements include high-quality data, structural pruning, and architectural reconfiguration.
- Small Language Models: MiniCPM combines hyper-parameter optimization, strategic training methodologies, architectural design, and high-quality data to enhance SLM capabilities.
- Scalable Pre-training Strategies: Scaling research has pursued scientifically predictable growth in LLMs, including training-stability techniques that preserve hyper-parameter consistency across model scales.
3 Model Wind Tunnel Experiments
Model Wind Tunnel Experiments use smaller models to investigate scalable training choices before transferring experience toward larger models. The experiments examine hyper-parameters, batch-size scaling, and learning-rate stability.
- Model Wind Tunnel Experiments: Model Wind Tunnel Experiments use SLMs to explore training limits before transferring experience to LLMs, covering hyper-parameters, batch-size scaling, and learning-rate stability.
- Scaling Hyper-parameters Invariant LM: Tensor Program width and depth scaling stabilize hyper-parameters across model sizes, and MiniCPM applies both techniques.
- Optimal Batch Size: Batch size balances convergence speed against computation: excessively large batches increase data and compute costs, while small batches require more steps and may limit loss reduction.
- Optimal Batch Size: The experiments track optimal batch-size trends with loss on C4.
- Optimal Batch Size: Figure 1 plots batch size against token consumption, colors loss, and represents each training curve as a horizontal color-point line.
- Optimal Batch Size: Figure 2 connects optimal batch sizes across the examined settings.
- Optimal Batch Size: As loss decreases, the optimal batch size shifts larger; the connected relationships are approximately linear in log space.
- Optimal Learning Rate: Tensor Program experiments find minimal learning-rate shifts during scaling, with the optimal base learning rate remaining around 0.01 despite a tenfold model-size increase.
4 WSD Learning Rate Scheduler
The Warmup-Stable-Decay learning-rate scheduler separates high-learning-rate training from decay, enabling continuous training and more efficient scaling-law measurement. Experiments show rapid loss reduction during decay, effective fixed-size model data scaling, and a higher data-to-model ratio than Chinchilla’s estimate.
- Cosine scheduler analysis: Cosine(T) with T = S achieves the lowest loss at training steps S = 20N, 40N, 60N, and 80N, whereas T < S and T > S are not optimal.The comparison uses Cosine(T) and CosineLoop(T) on a 0.036B model.
- Scheduler design: WSD LRS explicitly separates training into warmup, stable, and decay stages, with the learning rate decreasing during the remaining decay stage.The warmup ends at W, stable training ends at T, and the decay stage continues until S.
- Decay-stage dynamics: During WSD decay, loss rapidly decreases to equal or below the Cosine LRS loss at step T = S.The model can be reused before decay and continued with a high learning rate before annealing to reach comparable Cosine performance.
- Decay-stage dynamics: 10% of total tokens in the decay stage is sufficient for best results from stable checkpoints, while 2.5% falls short.This setting is used subsequently to ensure full convergence.
- Continuous data scaling: WSD enables continuous training of a fixed-size model toward extreme convergence and supports an optimal-loss envelope fitted as a power law in computation.The comparison continuously trains a 0.036B model against a 0.17B model with 40N data.
- Scaling-law measurement: WSD reduces scaling-law exploration from approximately O(m^2)C to O(mC) by avoiding retraining models from scratch at different data amounts.This makes optimal scaling properties more efficient to measure along the data axis.
5 Two Stage Pre-training Strategy
The strategy separates abundant coarse-quality pretraining from a decay phase that mixes high-quality knowledge and ability-oriented SFT data before a separate SFT stage. Experiments indicate that introducing high-quality data during decay improves the resulting model more than adding it only during SFT.
- Training Strategy: The proposed strategy uses large-scale coarse-quality data during pretraining, then mixes high-quality knowledge and ability-oriented SFT data into the decay phase.A separate SFT stage follows the decay phase.
- Ablation Study: Adding SFT data during decay improves results even when the subsequent SFT distribution is held constant.The comparison between A-2 and A-1 isolates the addition of SFT data during decay.
- Ablation Study: The weaker result from SFT-only training is not explained by insufficient SFT tokens.B-2 uses 12B SFT tokens, whereas B-3 uses 6B SFT tokens after high-quality data is introduced during annealing.
- Conclusion: The authors recommend beginning specialization and capability enhancement in the decay phase rather than waiting until SFT.They report that early introduction of high-quality data provides greater benefits than adding it only during SFT.
6 Model
MiniCPM combines compact architectures, shared embeddings, WSD-based three-stage training, curated data, and open-source evaluation. Its training loss drops sharply during decay, while benchmark results place the models strongly among SLMs and competitively against larger models.
- Architecture: MiniCPM uses 2.4B and 1.2B non-embedding parameter models with shared input-output embeddings to reduce the parameter footprint.The 1.2B model additionally uses Group Query Attention to reduce parameters.
- Training: The base model is trained through stable, decay, and SFT stages using Adam and the WSD learning-rate scheduler.Stable training uses approximately 1T data, while decay mixes pretraining and high-quality SFT data.
- Training Loss: During decay, the C4 loss decreases sharply and continues dropping after the learning rate falls below 10% of its maximum.The released model does not use the final decay checkpoints because SFT follows decay.
- Evaluation: The evaluation uses UltraEval with standardized prompts adapted to each model’s input-output template.The framework uses vLLM for inference and acceleration.
- Evaluation: MiniCPM-2.4B ranks highest among the evaluated SLMs on average and performs similarly to Mistral-7B-v0.1 in English while outperforming it in Chinese.MiniCPM-2.4B also outperforms Llama2-13B on most listed datasets, and MiniCPM-1.2B outperforms Llama2-7B except on HellaSwag.
7 MiniCPM Family
The MiniCPM family extends the base models through preference alignment, long-context training, and mixture-of-experts modeling. These variants show strong benchmark performance, including improved MTBench results, competitive 128K-context results, and sparse activation in the MoE model.
- MiniCPM-DPO: DPO raises MiniCPM-DPO-2.4B’s MTBench score from 6.89 after SFT to 7.25, surpassing Llama2-70B-Chat.The authors also observe a slight compromise on other benchmarks, termed the alignment tax.
- MiniCPM-128K: MiniCPM-2.4B-128K extends context length from 4,096 to 128,000 tokens for long-context reasoning.It reuses the stable-stage checkpoint and continues training on a mixture of short and long data.
- MiniCPM-128K: MiniCPM-2.4B-128K matches Mistral-7B-Instruct-v0.2 on ∞Bench and outperforms ChatGLM3-6B-128K despite being 2.5 times smaller.The benchmark includes long-context reasoning tasks beyond standard retrieval.
- MiniCPM-MoE: MiniCPM-MoE replaces dense MLP layers with eight experts and activates two experts per token.The model has 13.6B total non-embedding parameters and approximately 4B activated parameters.
- MiniCPM-MoE: The MiniCPM family is trained with WSD-based procedures, including a 130K-step pretraining and decay phase for the MoE model before benchmark evaluation.The MoE training uses a load-balancing loss to prevent collapse.
8 Conclusion
MiniCPM introduces compact models whose performance exceeds larger counterparts, supported by training methods that scale across model and data sizes. WSD enables continuous training, exposes training dynamics, and supports efficient scaling-law studies, while model wind-tunnel experiments explore transferable training choices.
- Conclusion: MiniCPM comprises 2.4B and 1.2B non-embedding parameter SLMs that demonstrate superior performance compared with larger counterparts.The paper presents their methods as potentially applicable to LLM development.
- Conclusion: The training methodologies scale across both model size and data size.This scalability is presented as a potential basis for future LLM research.
- Conclusion: WSD promotes continuous training, reveals distinctive training dynamics, and enables efficient study of scaling laws.The conclusion identifies WSD as a central contribution of the work.
Limitations
The paper identifies limits in validating its scaling-law analysis and in estimating optimal training settings. It also notes practical constraints and unresolved dependencies in the experimental methodology.
- Scope limitations: The scaling-law study covers SLMs but does not extend to LLM training, leaving WSD LRS application to LLMs unexplored.The authors remain optimistic about potential advantages, but do not claim validation at LLM scale.
- Batch-size estimation: The optimal batch-size estimate depends on a preliminary estimate of achievable loss informed by prior experiments.The paper characterizes this dependency as a “Chicken-and-egg” paradox and notes that more refined procedures may be developed.
- Optimization procedure: Optimal batch size and learning rate are likely correlated, so the study uses an iterative coordinate-descent-like procedure rather than a fully rigorous joint optimization.The authors explicitly welcome more rigorous methods in future work.
A.3 Model Architecture in Model Wind Tunnel Experiments
The model wind tunnel experiments standardize model shape and compare learning-rate schedules across training configurations. WSD LRS variants share a stable-training stage while differing in their decay endpoints.
- Model configuration: Model wind tunnel configurations keep model width relative to depth as similar as possible to reduce potential performance variation.Table 8 lists the model and training configurations used for the scaling curves.
- Learning-rate schedules: The study describes Cosine, CosineLoop, and WSD learning-rate schedulers for comparing training dynamics.An illustrative comparison between Cosine and WSD appears in Figure 15.
- Learning-rate schedules: WSD scheduler variants with different end steps share the same stable training stage before their decay stages diverge.This comparison isolates the effect of the decay endpoint across otherwise shared stable-training conditions.
B.2 Fitting the Data Scaling Law
The paper fits data scaling laws for continued training with WSD LRS and evaluates their behavior across models, tasks, compute, tokenization, and quantization settings. It reports polynomial fitting as the best form for continued training while documenting supporting experimental procedures.
- Data scaling law: Polynomial scaling law remains the best fit for continued training with WSD LRS, compared with an exponential alternative.Each Figure 16 point corresponds to the end of a WSD decay stage with a different end step.
- Data scaling law: Figure 17 plots the fitness of L(N, D) against real loss values along the data axis for each task and model.The fitted result is reported as satisfying except for the last checkpoints of the 0.11B and 0.25B models.
- Compute scaling: The compute-optimal regime is estimated by extracting Llama2 training-loss data and converting the axis to computation Flops.Figure 18 compares the resulting compute-based regime using straight lines for the optimal-loss envelope under WSD.
- Tokenization: The MiniCPM-120K tokenizer achieves the highest compression ratio on 300,000 held-out Chinese, English, code, and academic-paper documents.The evaluation documents were not part of the tokenizer’s training set.
- Quantization: MiniCPM applies 4-bit quantization to weight matrices while leaving embeddings and layer normalization unquantized because performance is sensitive to them.Weights are grouped along the input dimension and quantized separately; GPTQ calibration is used to reduce quantization loss.
E Edge Device Benchmarking
MiniCPM demonstrates inference on mobile and edge devices while emphasizing that the deployment work is a practicality demonstration rather than a mobile-optimization study. The reported examples show capabilities in translation, reasoning, instruction following, and structured outputs.
- Deployment scope: The mobile experiments demonstrate practical inference capability rather than optimized mobile deployment.The authors encourage further optimization and updates by the developer community.
- Device throughput: 18 tokens per second is achieved by MiniCPM-2.4B on an iPhone 15 Pro, with acceptable inference throughput on other tested devices.The model was adapted for Android, HarmonyOS, and iPhone systems and tested across various edge devices.
- Translation: MiniCPM-2.4B-DPO translates technical content better in the example, while both models handle an instruction trap and generate French without an intentionally included French corpus.Both models remain unnatural when expressing “just starting to learn Chinese.”
- Math and reasoning: Both models succeed at symbolic substitution and mathematical calculation in the presented math and reasoning examples.The passage reports example-level success rather than a benchmark-wide result.
- Instruction following: MiniCPM-2.4B-DPO follows tonal-pattern and acrostic instructions, whereas MiniCPM-1.2B follows only the tonal-pattern requirement.The comparison comes from the poem-generation example.
- Structured outputs: Both models understand emojis and structured data and control item counts implicitly, while the DPO model tends to generate more words.This is an example-based comparison of output behavior.