PaperlayerView exact revision on arXiv

Source-linked AI summary

Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation

Ling Team, Ang Li, Ben Liu, Binbin Hu, Bing Li, Bingwei Zeng, Borui Ye, Caizhi Tang, Changxin Tian, Chao Huang, Chao Zhang, Chen Qian, Chenchen Ju, Chenchen Li, Chengfu Tang, Chilin Fu, Chunshao Ren, Chunwei Wu, Cong Zhang, Cunyin Peng, Dafeng Xu, Daixin Wang, Dalong Zhang, Dingnan Jin, Dingyuan Zhu, Dongke Hu, Fangzheng Zhao, Feifan Wu, Feng Zhu, Gangshan Wang, Haitao Zhang, Hailin Zhao, Hanxiao Zhang, Hanzi Wang, Hao Qian, Haoyi Yu, Heng Zhang, Hongliang Zhang, Hongzhi Luan, Huirong Dong, Huizhong Li, Jia Li, Jia Liu, Jialong Zhu, Jian Sha, Jianping Wei, Jiaolong Yang, Jieyue Ma, Jiewei Wu, Jinjing Huang, Jingyun Tian, Jingyuan Zhang, Jinquan Sun, Juanhui Tu, Jun Liu, Jun Xu, Jun Zhou, Junjie Ou, Junpeng Fang, Kaihong Zhang, Kaiqin Hu, Ke Shi, Kun Tang, Kunlong Chen, Lanyin Mei, Lei Liang, Lei Xu, Libo Zhang, Lin Ju, Lin Yuan, Ling Zhong, Lintao Ma, Lu Liu, Lu Yu, Lun Cai, Meiqi Zhu, Mengying Li, Min Chen, Minghao Xue, Minghong Cai, Mingming Yin, Peijie Jiang, Peilong Zhao, Pingping Liu, Qian Zhao, Qing Cui, Qingxiang Huang, Qingyuan Yang, Quankun Yu, Shaowei Wei, Shijie Lian, Shoujian Zheng, Shun Song, Shungen Zhang, Shuo Zhang, Siyuan Li, Song Liu, Ting Guo, Tong Zhao, Wanli Gu, Weichang Wu, Weiguang Han, Wenjing Fang, Wubin Wang, Xiang Shu, Xiao Shi, Xiaoshun Lan, Xiaolu Zhang, Xiaqing Sun, Xin Zhao, Xingyu Lu, Xiong Xu, Xudong Wang, Xudong Wang, Xuemin Yang, Yajie Yang, Yang Xiang, Yanzhe Li, Yi Zhang, Yilong Wang, Yingxue Li, Yongzhen Guo, Yuzhuo Fu, Yuanyuan Wang, Yue Yang, Yue Yu, Yufeng Deng, Yun Zhang, Yunfei Yu, Yuqi Zhang, Yuxiao He, Zengke Gui, Zhaoxin Huan, Zhaoyang Wang, Zhibo Zhu, Zhihao Wang, Zhiqiang Zhang, Zhoufei Wang, Zihang Zeng, Ziqi Liu, Zitao Xuan, Zuoli Tang

arXiv:2510.22115v2cs.CLcs.AIcs.LG

TL;DR

Ling 2.0 addresses the challenge of improving reasoning while scaling language foundations efficiently. It combines sparse MoE architecture with coordinated training and infrastructure innovations, and reports strong multi-domain performance plus a Pareto frontier between reasoning accuracy and computational cost.

  • Problem

    As language models scale, performance gains increasingly depend on reasoning ability, while efficient scaling and sustained reasoning enhancement remain intertwined challenges.

  • Method

    Ling 2.0 combines high-sparsity MoE architecture with reasoning-oriented data, multi-stage alignment, evolutionary CoT training, scaling-law guidance, and FP8-based trillion-scale infrastructure.

  • Results

    Ling-1T outperforms leading models across multiple domains on most benchmarks and collectively the released models define a Pareto frontier between reasoning accuracy and computational cost.

  • Takeaways & Limitations

    Ling 2.0 provides a scalable foundation for general reasoning models, with sparse activation aligned to reasoning objectives across model scales.

  • Takeaways & Limitations

    The current GQA architecture constrains long-context efficiency, while reasoning depth, complex instruction following, and agentic behaviors remain under development.

Abstract

from arXiv · show

We introduce Ling 2.0, a series reasoning-oriented language foundation built upon the principle that every activation boosts reasoning capability. Designed to scale from tens of billions to one trillion parameters under a unified Mixture-of-Experts (MoE) paradigm, Ling 2.0 emphasizes high sparsity, cross-scale consistency, and efficiency guided by empirical scaling laws. The series includes three non-thinking (instruct) models - Ling-mini-2.0, Ling-flash-2.0, and Ling-1T - ranging from 16B to 1T total parameters and achieving up to 7-fold active-compute efficiency compared with dense counterparts. Ling 2.0 integrates coordinated innovations across model architecture, pre-training, post-training, and infrastructure: a high-sparsity MoE with MTP for efficient reasoning, reasoning-oriented data and mid-training CoT activation, reinforcement-based fine-tuning (DFT, Evo-CoT), and full-scale FP8 training with fine-grained heterogeneous pipelines. At the trillion scale, Ling-1T establishes a new Pareto frontier of reasoning accuracy versus computational efficiency, demonstrating that sparse activation, when properly aligned with reasoning objectives, enables scalable and efficient intelligence. Collectively, Ling 2.0 provides a coherent, open, and efficient foundation for advancing future reasoning and thinking models, including the Ring series built upon the same base.

1 Introduction

Ling 2.0 addresses trillion-parameter reasoning by coordinating sparse architecture, reasoning-focused training, alignment, and infrastructure innovations. Its released models follow a consistent scaling trajectory while targeting higher reasoning efficiency.

  • Motivation: Scaling trillion-parameter reasoning requires both computational efficiency and sustained, reliable reasoning improvement.The paper identifies dense-model cost, scaling prediction, infrastructure coordination, and cross-stage reasoning progress as intertwined challenges.
  • Approach: Ling 2.0 coordinates innovations across model architecture, pre-training, post-training, and infrastructure.The framework combines scaling laws, sparse MoE design, reasoning-oriented data, reinforcement-based training, and systems engineering.
  • Architecture: 7× efficiency leverage comes from activating 8 routed experts plus one shared expert among 256 experts per token.The high-sparsity MoE is scaled from 16B to 1T parameters and combines aux-loss-free balancing with multi-token prediction.
  • Pre-training: Reasoning-oriented data and progressive increases from 32% to 46% reasoning data support reasoning gains during 20T-token pre-training.Ling Math and Ling Code are prioritized, with the reported average reasoning-benchmark gain reaching 5–8%.
  • Post-training: Evo-CoT and sentence-level policy optimization deepen reasoning while improving training efficiency and stability.Evo-CoT reaches comparable or better competition-level mathematics performance with 25% fewer training tokens, while LPO reports around 10% improvements over token- and sequence-level baselines on complex reasoning benchmarks.
  • Results: Ling 2.0’s three released models span 16B to 1T total parameters, and performance improves steadily across mathematics, coding, reasoning, knowledge, alignment, and agentic tasks.Ling-mini-2.0 uses 1.4B activated parameters, Ling-flash-2.0 uses 6.1B, and Ling-1T uses 51B; Ling-1T establishes a reasoning-accuracy-versus-efficiency Pareto frontier.

2 Architecture

Ling 2.0 uses a unified, high-sparsity MoE architecture whose design choices are guided by scaling laws and validated across model and compute scales. The resulting configuration targets efficient training and inference while preserving model performance.

  • Model family: Ling 2.0 comprises three MoE models spanning 16B to 1T total parameters.The series includes Ling-mini-2.0, Ling-flash-2.0, and Ling-1T.
  • MoE architecture: Each model uses 256 routed experts, activates 8 experts plus one shared expert per token, and achieves approximately 3.5% activation.This high-sparsity, fine-grained configuration is shared across the Ling 2.0 series.
  • Training mechanisms: MTP and aux-loss-free load balancing are integrated to improve training efficiency, expert utilization, and performance on code and math tasks.Ling 2.0 uses one MTP layer per model scale with an MTP loss weight of 0.1.
  • Scaling-law design: Ling scaling laws indicate that MoE models use larger batch sizes and lower learning rates at larger compute scales, while optimal MoE models use fewer parameters and more training data than dense counterparts.These findings support near-optimal hyperparameter selection and data-model allocation for large-scale training.
  • MoE scaling laws: Scaling laws show that activation ratio primarily determines efficiency leverage, while expert granularity provides a nonlinear adjustment with an optimal range of 8–12 activated experts.Efficiency leverage also increases with training compute, whereas shared-expert and layer-arrangement choices have smaller effects.
  • MoE scaling laws: The unified efficiency-leverage law combines compute budget, activation ratio, and expert granularity, and is fitted and experimentally validated on Ling-mini-2.0.The law models compute-dependent amplification and granularity-dependent modulation of efficiency leverage.
  • Validation methodology: The Ling Wind Tunnel evaluates ideas across multiple model scales at 35% of the computational cost of a traditional ablation, producing more stable scaling conclusions.Its standardized pipeline supports extrapolation from smaller experiments toward much larger training regimes.

3 Pre-training

Ling 2.0 pre-training combines broad, quality-controlled data curation with reasoning-focused corpora, staged context expansion, and training procedures designed to improve efficiency and downstream reasoning performance.

  • Data and corpus construction: The pre-training data infrastructure covers general knowledge, code, mathematics, and multilingual content through specialized cleaning and quality-filtering pipelines.The pipelines combine rules, models, type-specific quality assessment, and automated rule generation.
  • Reasoning data: The Ling Code Corpus integrates source code, code-related natural language, and synthetic instructional data across 660 programming languages with quality stratification and validation.Lint-based checks remove compilation or structural errors, while rephrasing and paraphrasing augment the corpus.
  • Reasoning data: The Ling math corpus outperformed competitive baselines in 1B-model validation, while Ling’s curated mathematical web data surpassed several open-source datasets under the same training paradigm.The web-data comparison included Infi-mm-math, finemath-3plus, megamath, and nemotron-cc.
  • Data and corpus construction: The data pipeline yields approximately 1.2 T high-quality long-text tokens and processes 30 billion trainable data points in two days.The infrastructure also supports a trillion-record web-wide table and automated data exploration.
  • Training recipe: WSM replaces learning-rate decay with checkpoint merging, achieves +1 to +2 leaderboard-point average gains, and retains its advantage after supervised fine-tuning.Unlike WSD, WSM requires no preset decay start or decay data budget.

4 Post-Training

Ling 2.0’s post-training pipeline combines decoupled supervision, evolutionary reasoning, task-specific rewards, and scalable evaluation infrastructure to strengthen reasoning while preserving efficiency. Across its model series, these methods support broad gains on coding, mathematics, and reasoning benchmarks, with Ling-1T showing strong multi-domain performance.

  • Supervised Fine-Tuning: DFT uses differentiated system prompts to create instant-response and in-depth-reasoning modes, establishing a foundation for reinforcement learning.The SFT data combines reasoning, general, and industrial tasks to support transitions between abstract reasoning and practical problem solving.
  • Evolutionary Reasoning Reinforcement Learning: Evo-CoT progressively increases reasoning depth according to problem complexity, while sustaining reasoning on complex scenarios and efficiency on general tasks.The method builds on the DFT-initialized policy and is paired with linguistic-unit optimization for more stable reasoning updates.
  • Task-Specific Rewards: Task-specific rewards balance correctness, clarity, and efficiency across code, front-end, and other reasoning tasks.VAR evaluates rendered interfaces with a multimodal model, while length preferences encourage concise outputs on easier tasks and more extensive reasoning on harder tasks.
  • Reward Modeling: GAR and RubriX address noisy subjective evaluation through group comparison and fine-grained, multidimensional preference rubrics.RubriX covers dimensions such as clarity, coherence, creativity, emotional resonance, instruction adherence, and domain-specific accuracy.
  • Reward Infrastructure: The unified reward model system supports rule-based, model-based, and multilingual-program verification at 40K concurrent requests with success rates exceeding 99.9%.ApexEval selects promising checkpoints for subsequent instruction tuning or reinforcement-learning optimization.
  • Evaluation Results: Ling-mini-2.0, Ling-flash-2.0, and Ling-1T outperform most leading industry models across coding, mathematics, and reasoning benchmarks, while Ling-1T performs strongly across multiple domains.The reported comparisons include LiveCodeBench, MBPP Sanitized, CodeForces, CNMO 2024, Omni-MATH, OptMATH, BBEH, KOR-Bench, and ZebraLogic.

5 Infrastructure

Ling 2.0 pairs sparse MoE architecture with precision-aware and heterogeneous infrastructure to make trillion-scale training more efficient. The section reports throughput, memory, communication, checkpoint, and startup optimizations, while noting sensitivity to routing and pipeline balance.

  • Infrastructure rationale: High-sparsity MoE requires matching infrastructure because, without engineering optimizations, it offers no performance advantage over dense models.The architecture’s benefits depend on coordinated algorithm–system design.
  • FP8 training: FP8 training achieves near-lossless performance while reducing computation and memory consumption.The framework uses fine-grained quantization and safeguards to manage precision and stability.
  • FP8 training: 15% MFU improvement over BF16 training is achieved by Ling-1T, while Ling-mini-2.0 improves throughput by 30-60% with MTP and 90-120% without MTP.These comparisons are reported on 8/16/32 80GB GPUs against LLaMA 3.1 8B and Qwen3 8B.
  • Heterogeneous pipeline: 40% relative end-to-end improvement comes from refining pipeline partitioning around MTP’s approximately 1.7× standard-MoE-layer computation cost.Balanced routing can add 5% through more virtual pipeline stages, but imbalanced routing may eliminate end-to-end gains.
  • Communication optimization: 13% end-to-end performance improvement comes from operator fusion, while reducing communication redundancy adds 2% during Ling-1T training.These gains use DeepEP for intra-node operations despite avoiding cross-node expert-parallel communication.
  • Performance boundary: End-to-end gains vary with model scale: smaller models achieve up to 10%, whereas larger models achieve approximately 7%.Persistent pipeline bottlenecks can overlap away recomputation gains.
  • Framework optimization: Checkpoint metadata caching reduces Ling-1T save time from 269s to 30s, and startup warm-up reduces the first training step by approximately 30%.The checkpoint optimization also reduces checkpoint saving’s share of total training time from 2.43% to 0.82%.

5.4 Software Engineering for Foundation LLMs

The paper treats foundation-model development as a software-engineering problem involving costly, unpredictable, heterogeneous, and tightly coupled algorithm–system decisions. Ling 2.0 applies the 4C principle through estimation, monitoring, infrastructure-aware design, cross-platform alignment, and continuous evaluation.

  • Motivation: Foundation LLM development involves substantial cost and complexity, with testing constrained by resource limits and defects often emerging late.The paper frames outcome predictability and early risk detection as essential.
  • Motivation: Heterogeneous hardware differs in precision and performance, making alignment of model behavior and efficiency a continuing research challenge.The paper identifies algorithm–engineering trade-offs as part of this deployment complexity.
  • 4C principle: The 4C principle—Correct, Consistent, Complete, and Co-Design—aims to improve development efficiency and delivery quality while reducing costs.Ling 2.0 uses the principle as a framework for software-engineering practices.
  • Iterative development: A progressive estimation, release approval, monitoring, sampling, and experience-accumulation workflow uses up to approximately 3% of actual training resources per iteration.The workflow supports repeated validation during long training cycles.
  • Co-design: Infrastructure-aware design rejects techniques whose computation and memory overhead outweigh their performance gains, such as a Norm Head improvement of less than 1%.The paper also configures distributed parameters to reduce pipeline bubbles across heterogeneous modules.
  • Cross-platform alignment: Loss differences across GPU architectures oscillate around zero, with mean errors within one-thousandth, supporting training-convergence alignment.The result is reported for Ling-flash-2.0.
  • Continuous evaluation: Distributed inference, prompt reuse, and asynchronous batching allow checkpoint validation within hours rather than days.These optimizations make evaluation a continuous feedback component of training.
  • Computation–communication overlap: Large expert-parallel overlap is sensitive to routing imbalance: shallow-layer imbalance can cause OOM errors and force a performance-costly pipeline repartition.The paper therefore identifies routing and related components as targets for joint optimization.

6 Conclusion

Ling 2.0 argues that coordinated architectural, training, and infrastructure innovations support scalable sparse reasoning foundations. It also identifies remaining limits in long-context efficiency, reasoning depth, instruction following, and agentic behavior.

  • Conclusion: Ling 2.0 combines high-sparsity MoE, reasoning-oriented data, multi-stage alignment, and FP8 infrastructure to establish a scalable foundation for general reasoning models.The conclusion presents these as coordinated innovations across architecture, training, and infrastructure.
  • Open challenges: Current GQA architecture constrains long-context efficiency, while reasoning length and depth, complex instruction following, and agentic behaviors remain open challenges.Future work targets linear or sparse attention and more general, autonomous, interactive capabilities.

7 Contributors

The contributor section lists the Ling Team and individual contributors, identifies corresponding authors with a dagger, and states that authors are listed alphabetically by first name.

  • Authorship: Authors are listed alphabetically by the first name.
  • Contributors: The contributor list includes the Ling Team and the named individual authors.The names are distributed across the contributor passages.
  • Authorship: A dagger denotes corresponding authors.

A The Method to Compute Save Interval

The method models checkpoint-storage overhead and failover rollback cost as a function of the saving interval, then derives the interval minimizing ETTR impact.

  • A The Method to Compute Save Interval: Checkpoint storage overhead and failover rollback cost are expressed as components of the ETTR impact.C denotes per-checkpoint storage overhead, while F and A represent daily failover events and the time cost of each failover.
  • A The Method to Compute Save Interval: Removing constant terms reduces the optimization to the interval-dependent checkpoint-saving expression.The derivation retains the trade-off between checkpoint frequency and storage overhead while treating F ∗A as constant.
  • A The Method to Compute Save Interval: s = q(2880∗C/F) gives the optimal saving interval corresponding to minimal failover impact on ETTR.In Ling-1T training, the configured checkpoint saving interval is 48 minutes.

B.1.1 Ling Code Corpus

The Ling Code Corpus combines broad, quality-stratified code and code-related data with synthetic augmentation, and validates its usefulness through small-model experiments.

  • B.1.1 Ling Code Corpus: The Ling Code Corpus integrates source code, code-related natural language, and synthetic instructional data across programming languages and domains.Its curation emphasizes breadth of coverage and depth of quality control.
  • B.1.1 Ling Code Corpus: 2.7 T tokens of deduplicated source code cover 660 programming languages before quality stratification.The stratification dimensions are code style and readability, norm adherence and structure, and complexity and difficulty.
  • B.1.1 Ling Code Corpus: 600 B tokens form the top-quality curated code subset, followed by 300 B tokens of rephrased and paraphrased code augmentation.The additional augmentation targets linguistic diversity and naturalness.
  • B.1.1 Ling Code Corpus: 140 B+ tokens of high-quality refined code-related Common Crawl data remain after filtering approximately 700 B collected tokens.The collection uses two-stage recall over code-related pages, tutorials, and developer discussions.
  • B.1.1 Ling Code Corpus: The corpus also includes 73 B tokens of reconstructed commit-level records, programming-contest data, and a small diverse synthetic component.Commit reconstruction captures developer intent, revision rationale, and contextual discussions.
  • B.1.1 Ling Code Corpus: Training 1B coding models from scratch on single-type code data provides a reliable proxy for full-scale performance and supports early validation.The resulting base models show strong task competence and consistent behavioral correlation with larger-scale models.

B.1.2 Ling Math Corpus

The Ling Math Corpus exceeds 1.8T tokens and combines multi-source curation with synthetic reasoning data, while 1B-model evaluations report advantages over competitive and open-source alternatives.

  • B.1.2 Ling Math Corpus: The corpus exceeds 1.8T mathematics tokens from web pages, textbooks, research papers, code repositories, problem banks, and synthetic sources.Parsing, recall, filtering, rewriting, synthesis, and balancing produce the refined corpus.
  • B.1.2 Ling Math Corpus: A multi-stage recall pipeline uses fastText retrieval, 4B-parameter LLM-Filter and LLM-Refiner models, and deduplication to curate mathematical content.The pipeline targets mathematical knowledge and step-by-step problem-solving processes.
  • B.1.2 Ling Math Corpus: Synthetic generation adds varied-difficulty mathematical Q&A pairs with step-by-step reasoning, alongside extracted and newly synthesized problems.New problems are synthesized from a large-scale mathematical concept graph.
  • B.1.2 Ling Math Corpus: The Ling-math-1B model is trained on math-only data for over 1.8T tokens, with the final 300B used for annealing.Evaluation reports average benchmark performance across mainstream mathematical tasks.
  • B.1.2 Ling Math Corpus: Ling-math-1B outperforms Qwen2.5-Math-1.5B-Base and Qwen3-1.7B-Base on mainstream mathematical benchmarks.The reported benchmarks include GSM8K, MATH, CollegeMath, OlympiadBench, CMATH, and MathBench.
  • B.1.2 Ling Math Corpus: Ling-math-web-1B trained on the curated web data shows markedly superior performance to several open-source mathematical datasets.The comparison includes Infi-mm-math, finemath-3plus, megamath, and nemotron-cc.

B.2 Multilingual Data

The multilingual corpus combines open-source web, parallel, code, mathematics, and other domain data across about 30 languages, with distribution choices affecting language-specific performance.

  • B.2 Multilingual Data: The multilingual corpus contains 2TB of data from open-source web datasets and classical parallel corpora supporting cross-lingual alignment.Sources include CulturaX, WanJuan, OPUS, and MultiUN.
  • B.2 Multilingual Data: Coverage spans about 30 languages across web pages, code, mathematics, Wikipedia, parallel corpora, and synthetic translation data.The initial set includes 18 individual languages spanning Germanic, Romance, Slavic, Vietnamese, Thai, Korean, and Indonesian groups.
  • B.2 Multilingual Data: Additional mixed-language corpora include Japanese, Arabic, Hindi, Turkish, and Finnish.These languages supplement the explicitly listed initial language set.
  • B.2 Multilingual Data: The distribution maintains Chinese and English performance while significantly improving minor-language performance.Romance and Germanic data have less negative effect on English and Chinese benchmarks than lower-quality Slavic and other-language data.

B.3 Data Infrastructure

The data infrastructure addresses inefficient collaboration, opaque lineage, slow iteration, and data silos through Data-as-Code and a Unified Data Lakehouse. Together, these principles support reproducible workflows, centralized data operations, and large-scale corpus construction.

  • The infrastructure is built on two principles: Data-as-Code and a Unified Data Lakehouse.
  • Data-as-Code: Data-as-Code codifies the entire data-processing pipeline in version control to enable automated and reproducible workflows.
  • Data-as-Code: A unified AIDataOps library centralizes over 50 multimodal data-processing operators within an automated CI/CD system.This supports end-to-end transparent and reproducible data lineage.
  • Unified Data Lakehouse: The lakehouse consolidates disparate sources through a unified logical wide table, addressing data silos across hundreds of independent datasets.
  • Unified Data Lakehouse: The resulting data engine enabled a wide table containing trillions of web-data records and processed 30 billion trainable data points in two days.It was instrumental in building the Ling 2.0 corpus and supports further data exploration.
Loading 2510.22115v2…