PaperlayerView exact revision on arXiv

Source-linked AI summary

PCMind-2.1-Kaiyuan-2B Technical Report

Kairong Luo, Zhenbo Sun, Xinyu Shi, Shengqi Chen, Bowen Yu, Yunyi Chen, Chenyi Dang, Hengtao Tao, Hui Wang, Fangming Liu, Kaifeng Lyu, Wenguang Chen

arXiv:2512.07612v1cs.CLcs.AIcs.LG

TL;DR

Open-source pretraining lacks transparent data and training recipes, while heterogeneous datasets and limited compute complicate efficient model development. Kaiyuan-2B addresses these constraints with quantile benchmarking, selective repetition, quality-based curriculum training, and FP16-stable architecture changes. The resulting fully open-source model reaches the frontier of comparable models and provides a transparent, resource-efficient pretraining framework.

  • Problem

    Open-source communities face closed industry data and training recipes, heterogeneous datasets, and limited compute when trying to match open-weight models.

  • Method

    Kaiyuan-2B combines Quantile Data Benchmarking, multi-phase selective repetition, quality-based curriculum training, and architectural changes for FP16 stability.

  • Results

    Kaiyuan-2B achieves frontier performance among fully open-source models at comparable scales and approaches or matches larger models on several capabilities.

  • Takeaways & Limitations

    The project provides a practical, transparent framework for selecting and using public data under resource constraints, releasing its model, preprocessing framework, and pretraining dataset.

  • Takeaways & Limitations

    The dataset analysis remains primarily qualitative and coarse-grained, leaving finer-grained comparison and mixing-ratio optimization for future work.

Abstract

from arXiv · show

The rapid advancement of Large Language Models (LLMs) has resulted in a significant knowledge gap between the open-source community and industry, primarily because the latter relies on closed-source, high-quality data and training recipes. To address this, we introduce PCMind-2.1-Kaiyuan-2B, a fully open-source 2-billion-parameter model focused on improving training efficiency and effectiveness under resource constraints. Our methodology includes three key innovations: a Quantile Data Benchmarking method for systematically comparing heterogeneous open-source datasets and providing insights on data mixing strategies; a Strategic Selective Repetition scheme within a multi-phase paradigm to effectively leverage sparse, high-quality data; and a Multi-Domain Curriculum Training policy that orders samples by quality. Supported by a highly optimized data preprocessing pipeline and architectural modifications for FP16 stability, Kaiyuan-2B achieves performance competitive with state-of-the-art fully open-source models, demonstrating practical and scalable solutions for resource-limited pretraining. We release all assets (including model weights, data, and code) under Apache 2.0 license at https://huggingface.co/thu-pacman/PCMind-2.1-Kaiyuan-2B.

1 Introduction

Kaiyuan-2B addresses heterogeneous open-source data and limited compute through data benchmarking, selective repetition, and quality-based curriculum training. The fully open-source 2B model reaches the frontier of comparable fully open-source models while approaching larger open-weight models.

  • Motivation: Open-source pretraining is constrained by closed industry data and recipes, heterogeneous datasets, and compute budgets far below industry-scale token counts.These constraints make dataset comparison, selection, mixing, and efficient use of high-quality data central challenges.
  • Deduplication and Quantile Data Benchmarking: Quantile Data Benchmarking compares heterogeneous datasets by selecting quality-score quantiles and measuring their characteristics with a small reference model’s downstream performance.Deduplication precedes benchmarking, which supports systematic comparisons across leading datasets and their internal partitions.
  • Strategic Selective Repetition for Sparse High-Quality Data: Selective repetition uses a five-phase training pipeline in which high-quality samples can recur across phases while ordinary samples appear once per phase.The design increases exposure to higher-quality data while limiting overall repetition benefits to levels similar to one-pass training regimes [53] [77].
  • Multi-Domain Curriculum Training: Multi-Domain Curriculum Training keeps dataset mixtures stable while ordering samples by quality within each dataset, presenting more important data later in phases 3–5.Datasets without explicit quality labels are shuffled, and moderate learning-rate decay plus model averaging are used to exploit the curriculum.
  • Pushing the Frontier of Fully Open-Source Models: Kaiyuan-2B substantially outperforms Gemma2-2B at a similar parameter scale and approaches or matches larger models on reasoning and knowledge-intensive tasks.Its efficiency advantages become more pronounced when comparing only 1.4B non-embedding parameters.

2 Architecture Design and Training Stability

Kaiyuan-2B targets FP16 instability on Ascend 910A hardware by identifying overflow-prone activations and modifying the architecture to bound them. Logits soft-capping and sandwich normalization reduce activation magnitudes and support stable training.

  • Hardware and Precision: Kaiyuan-2B uses Ascend 910A accelerators that support only FP16, motivating architectural changes for numerical stability despite mixed-precision execution.Softmax and RMSNorm run in FP32, while remaining computations run in FP16.
  • Instability Sources: FP16 training becomes unstable because attention logits and post-SwiGLU activations grow excessively, exceeding 10,000 after one trillion tokens.The growth approaches the FP16 upper limit and forces the dynamic loss scaler to reduce its scaling factor.
  • Architectural Optimization: Logits soft-capping reduces the L1 norm of attention logits by about an order of magnitude, while sandwich normalization keeps MLP activations within a safe range.Both techniques place strict bounds on activation values and reduce large-value accumulation in residual connections.

3 Data Benchmarking and Preprocessing

The section introduces quantile benchmarking to compare heterogeneous datasets empirically across quality ranges and target capabilities, then uses the findings to guide data selection, mixing, and curriculum strategies.

  • 3.1.1 Quantile Benchmarking Methodology: Quantile benchmarking trains small reference models on quality-stratified subsets and evaluates them across benchmarks to compare dataset characteristics.The process samples fixed-size subsets around quality-score quantiles and can evaluate models from scratch or by continual training.
  • 3.1.3 Implications for Data Selection: Quantile comparisons provide guidance for dataset selection and mixing, motivating later-stage scheduling and higher repetition of high-quality partitions.The proposed curriculum places refined samples later in training and applies selective multi-epoch training instead of repeating entire datasets.
  • 3.1.2 Benchmarking Results and Analysis: Fineweb-Edu performs better on knowledge-intensive and formal-semantic tasks, whereas DCLM-Baseline has advantages on situated commonsense reasoning.Figure 4 represents the contrast on MMLU and WinoGrande, while broader comparisons include CSQA, BoolQ, PIQA, Social IQa, and HellaSwag.
  • 3.1.2 Benchmarking Results and Analysis: DCLM-Baseline shows substantial within-dataset variation, including 2% on MMLU and 8% on ARC-Easy between top 0% and top 60% quantiles.These differences occur in the continual-training scenario and demonstrate that quality varies considerably within a single dataset.
  • 3.1.2 Benchmarking Results and Analysis: Benchmark performance is not always monotonic with quality scores, since higher-scoring partitions can reduce HellaSwag and PIQA performance.The result indicates that dataset quality metrics are task-specific rather than universally predictive across capabilities.
  • 3.1.3 Implications for Data Selection: The current dataset comparison and mixing analysis remains qualitative and coarse-grained, leaving finer quantitative optimization for future work.This limitation bounds how precisely the benchmarking results can determine mixing ratios.

4 Multi-Phase Multi-Dataset Curriculum Training

Kaiyuan-2B uses a five-phase curriculum that progressively shifts domain mixtures, selectively repeats high-quality data, and orders samples by quality while maintaining stable dataset proportions. Experiments support mild repetition and curriculum model averaging, while training curves reveal validation instability at phase transitions and motivate more gradual transitions.

  • Selective Repetition: High-quality samples are exposed later and repeated across phases, with progressively smaller top-k retention ratios rather than repeating entire datasets.Each phase mostly presents each retained sample once, concentrating repetition on high-quality partitions.
  • Domain-Level Curriculum: Kaiyuan-2B progressively increases Chinese, code, and mathematical data in later phases while keeping English above 30% and each specialized domain below 30%.Supervised fine-tuning data is introduced in the final two phases, with phase mixtures specified in Tables 6–10.
  • Selective Repetition: Retaining 33.4% of top-quality samples for three epochs outperforms one-pass training in a 1.5B Qwen2.5 experiment on 30B DCLM-Baseline tokens.The experiment supports mild strategic repetition as a more efficient alternative to one-pass training.
  • Multi-Dataset Curriculum: The multi-dataset curriculum ranks samples within each dataset, rescales ranks globally, and interleaves them to preserve quality ordering, mixture proportions, and stable dataset mixtures.Datasets without quality labels are shuffled, and the curriculum mixture is informed by quantile benchmarking.
  • Training Stability: Each phase shows accelerated validation-loss decay followed by sharp increases, likely reflecting high-quality-data benefits and curriculum-induced distribution shifts.The authors suggest more diverse validation sets, more gradual domain transitions, and more phases for improved stability.

5 Evaluation

Kaiyuan-2B is evaluated across mathematics, coding, Chinese, and reasoning and knowledge benchmarks against comparable open-weight and fully open models. It outperforms similarly sized fully open baselines, remains competitive with larger models, and narrows the gap to leading open-weight systems.

  • Evaluation Setup: The evaluation compares Kaiyuan-2B with open-weight and fully open base models using generation for math and coding and perplexity-based evaluation for other domains.The benchmark suite covers four domains, with PPL tasks assessed using multiple-choice and completion formulations and the superior score reported.
  • Core Capabilities: Math, Code, and Chinese.: Kaiyuan-2B achieves a 46.05 average across Chinese, math, and coding benchmarks, outperforming similarly sized fully open models while remaining competitive with larger models.It scores 46.30 on C-Eval, 49.25 on CMMLU, 51.33 on GSM8K, 30.34 on MATH, and 42.68 on HumanEval.
  • Reasoning and Knowledge.: Kaiyuan-2B achieves a 67.74 average on reasoning and knowledge benchmarks, surpassing SmolLM2-1.7B by +1.69 and OLMo-2-0425-1B by +5.68 while matching YuLan-Mini-2.4B.SmolLM3-3B scores 72.60, while Kaiyuan-2B approaches Gemma2-2B at 69.16 and trails Qwen3-4B at 81.84.
  • Discussion on Size and Performance Trade-offs.: Kaiyuan-2B lies beyond the current fully open frontier, outperforming earlier fully open models at comparable parameter counts and approaching YuLan-Mini-2.4B.The comparison also reports model statistics using both total and non-embedding parameters.
  • Discussion on Size and Performance Trade-offs.: Kaiyuan-2B uses 1.4B non-embedding and 2B total parameters versus Gemma2-2B’s 2B non-embedding and 2.6B total parameters, with similar token budgets.The report presents this comparison as evidence of a favorable size-performance trade-off for resource-constrained environments.

6 Conclusion

The report presents Kaiyuan-2B as a systematic, resource-efficient approach to fully open-source pretraining. Its contributions combine quantile benchmarking, strategic repetition, and multi-domain curriculum training with complete release of the model, data-processing framework, and final dataset.

  • 6 Conclusion: Kaiyuan-2B provides a systematic and resource-efficient approach to fully open-source pretraining under data heterogeneity and computational scarcity.The project frames its approach as providing concrete answers to both challenges.
  • 6 Conclusion: The core contributions are Quantile Data Benchmarking, Strategic Manual Repetition, and Multi-Domain Curriculum Training.Together, these methods form the report’s practical framework for selecting and using public data effectively.
  • 6 Conclusion: The project releases the model checkpoint, open-source data preprocessing framework, and final pretraining dataset to support transparent reproduction and further research.

A Quality-Score Quantile Benchmarking

The report uses quantile benchmarking to compare heterogeneous open-source datasets through reference-model experiments on quality-selected subsets. The reported benchmark figures distinguish DCLM-Baseline as stronger for understanding-oriented benchmarks and FineWeb-Edu as stronger for knowledge-oriented benchmarks.

  • A Quality-Score Quantile Benchmarking: Quantile benchmarking evaluates dominant datasets by training reference models on subsets selected around target quality-score quantiles.The method is applied to datasets such as DCLM-Baseline and FineWeb-Edu, with experiments conducted from scratch and from pretrained checkpoints.
  • A Quality-Score Quantile Benchmarking: DCLM-Baseline is better on understanding-oriented benchmarks, whereas FineWeb-Edu is better on knowledge-oriented benchmarks.

B Datasets Used in Training

The training corpus is assembled from publicly available datasets listed in Table 5, with licensing reviewed for release under a permissive license. Datasets that prohibit redistribution, such as the cited Nemotron series data, are excluded.

  • B Datasets Used in Training: Table 5 lists all datasets used to train PCMind-2.1-Kaiyuan-2B, and the datasets are publicly available to acquire.Most are hosted on Hugging Face unless otherwise noted.
  • B Datasets Used in Training: The authors screen dataset licenses and agreements to support releasing the model and training data under a permissive license on an as-is, use-at-your-own-risk basis.The same datasets are intended to remain available for reproduction.
  • B Datasets Used in Training: Nemotron series datasets are excluded because the NVIDIA Data Agreement for Model Training disallows redistribution and public display.

C Phase-wise Data Mixture

The multi-phase mixture combines full, quality-filtered, randomly sampled, and repeated datasets across English, Chinese, code, math, and SFT. Training stability and reproducibility are supported by bounded activations, explicit hyperparameters, and near-end checkpoint averaging.

  • C Phase-wise Data Mixture: The five phases combine full datasets, quality-filtered subsets, random samples, and repeated data across English, Chinese, code, math, and SFT.Dataset counts and detailed compositions are provided for Phases 1–5.
  • C Phase-wise Data Mixture: Quality filtering retains top-scoring samples, while datasets without quality metrics are randomly sampled and repeated datasets can exceed a 100.0% actual ratio.Examples include top-score retention for Fineweb-Edu-CN and StarCoder, random sampling for StarCoder and LLM360-Math, and repeated FineWiki-CN.
  • D.1 Implementation of Stability Components: Logits soft-capping bounds language-model output logits before cross-entropy, while Sandwich Normalization normalizes residual branches to limit activation growth.The implementation uses a soft-capping threshold of 30.0 and applies normalization on residual branches.
  • Training Configuration: The training configuration specifies AdamW, phase-dependent learning rates, dynamic loss scaling for FP16, and averaging of near-end checkpoints.Checkpoint evaluations found fluctuations without a clear trend, motivating simple model averaging.

D.4 Reference Experiments for Quantile Benchmarking

The reference experiments test quantile-selected data in both training-from-scratch and continual-training settings, then compare efficient high-quality-data strategies. Results align across scenarios and quantify the benchmarking compute budget as small relative to full pretraining.

  • Consistency Across Scenarios: Quantile benchmarking produces strongly aligned results for training from scratch and continual training on both DCLM-Baseline and Fineweb-Edu.The alignment persists even when continual training resumes from a checkpoint trained only on deduplicated DCLM-Baseline, supporting robustness across paradigms.
  • Compute Cost Comparison: The 0.6B model trained on 42B tokens uses approximately 2.07% of the reference compute budget and less than 0.6% of the total pretraining budget.The estimate uses C = 6ND with DCLM-Baseline as the reference.
  • High-Quality Data Utilization Strategies: The experiments compare uniform-order baseline training with high-quality-data repetition and curriculum training combined with model averaging.Repetition matches the baseline’s computational FLOPs, while curriculum variants use moderate or no learning-rate decay and average final checkpoints.
  • Evaluation Settings: Evaluation uses the Core subset containing MMLU [31], ARC [13], and CSQA [70] because these benchmarks discriminate performance differences between training approaches.Curriculum experiments use a higher final learning rate of 1 × 10^-3 and exponential moving averaging over the final six checkpoints.

E Model Performance across Benchmarks (Full Table)

The full benchmark table evaluates models across target capability dimensions and is paired with performance–parameter visualizations. Kaiyuan-2B is reported to lie on the frontier of fully open-source models.

  • E Model Performance across Benchmarks (Full Table): Kaiyuan-2B lies on the performance–parameter frontier of fully open-source models across the reported benchmark dimensions.Table 17 merges results from Tables 2 and 3, while Figures 1 and 8 visualize performance against parameter counts.
  • E Model Performance across Benchmarks (Full Table): Table 17 provides a complete evaluation across the target capability dimensions while accounting for differences in total and non-embedding parameters.The accompanying visualizations make these parameter differences explicit.
  • E Model Performance across Benchmarks (Full Table): The report uses CMA for both CMA and CDMA variants, with the Table 1 CMA method specifically corresponding to CDMA with learning-rate decay.This naming convention applies to the reported comparison context.
Loading 2512.07612v1…