PaperlayerView exact revision on arXiv

Source-linked AI summary

Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs

Yichun Yin, Wenyong Huang, Kaikai Song, Yehui Tang, Xueyu Wu, Wei Guo, Peng Guo, Yaoyuan Wang, Xiaojun Meng, Yasheng Wang, Dong Li, Can Chen, Dandan Tu, Yin Li, Fisher Yu, Ruiming Tang, Yunhe Wang, Baojun Wang, Bin Wang, Bo Wang, Boxiao Liu, Changzheng Zhang, Duyu Tang, Fei Mi, Hui Jin, Jiansheng Wei, Jiarui Qin, Jinpeng Li, Jun Zhao, Liqun Deng, Lin Li, Minghui Xu, Naifu Zhang, Nianzu Zheng, Qiang Li, Rongju Ruan, Shengjun Cheng, Tianyu Guo, Wei He, Wei Li, Weiwen Liu, Wulong Liu, Xinyi Dai, Yonghan Dong, Yu Pan, Yue Li, Yufei Wang, Yujun Li, Yunsheng Ni, Zhe Liu, Zhenhe Zhang, Zhicheng Liu

arXiv:2504.07866v2cs.CLcs.AI

TL;DR

Training dense language models above 100 billion parameters presents stability and system-efficiency challenges. Pangu Ultra combines depth-scaled sandwich normalization with large-scale training and system optimizations; it outperforms leading dense models on almost all major language tasks and is competitive with larger sparse models.

  • Problem

    Training deep, hundred-billion-parameter dense models poses stability challenges, including loss spikes, as well as system-efficiency challenges requiring thousands of processors.

    Source: §1 · p. 1

  • Method

    Pangu Ultra is a 135-billion-parameter dense Transformer trained with depth-scaled sandwich norm and TinyInit, followed by post-training to enhance reasoning.

    Source: §1 · p. 2, §6 · p. 14

  • Results

    Pangu Ultra surpasses Llama 405B and Mistral Large 2 on almost all major language tasks and achieves competitive performance against larger sparse models such as DeepSeek-R1.

    Source: §6 · p. 15

  • Takeaways & Limitations

    The authors conclude that Ascend NPUs can train dense models with hundreds of billions of parameters.

    Source: §6 · p. 15

Abstract

from arXiv · show

We present Pangu Ultra, a Large Language Model (LLM) with 135 billion parameters and dense Transformer modules trained on Ascend Neural Processing Units (NPUs). Although the field of LLM has been witnessing unprecedented advances in pushing the scale and capability of LLM in recent years, training such a large-scale model still involves significant optimization and system challenges. To stabilize the training process, we propose depth-scaled sandwich normalization, which effectively eliminates loss spikes during the training process of deep models. We pre-train our model on 13.2 trillion diverse and high-quality tokens and further enhance its reasoning capabilities during post-training. To perform such large-scale training efficiently, we utilize 8,192 Ascend NPUs with a series of system optimizations. Evaluations on multiple diverse benchmarks indicate that Pangu Ultra significantly advances the state-of-the-art capabilities of dense LLMs such as Llama 405B and Mistral Large 2, and even achieves competitive results with DeepSeek-R1, whose sparse model structure contains much more parameters. Our exploration demonstrates that Ascend NPUs are capable of efficiently and effectively training dense models with more than 100 billion parameters. Our model and system will be available for our commercial customers.

1 Introduction

Pangu Ultra explores whether dense models can scale beyond 100 billion parameters while remaining competitive with large sparse models. It combines training-stability techniques, a multi-stage training pipeline, and system optimizations; on public benchmarks it outperforms leading dense models on almost all major language tasks and competes with sparse models above 500 billion parameters.

1 Introduction

  • The study targets dense models at large scales because their parameters are generally easier to optimize and their deterministic inference-time parameter use simplifies system optimization.The authors aim to show dense-model performance can match state-of-the-art MoE models across diverse tasks.Source: §1 · p. 1
  • Training deep models can produce loss spikes, while orchestrating thousands of processors for dense models above 100 billion parameters creates significant system-efficiency challenges.The introduction links loss spikes to potential damage to the training process.Source: §1 · p. 1
  • Pangu Ultra is a 135-billion-parameter, 94-layer dense Transformer; depth-scaled sandwich norm and TinyInit are designed to stabilize gradients and eliminate loss spikes during training.The authors attribute loss spikes to gradient fluctuations that may hinder convergence or lead to divergence.Source: §1 · p. 2
  • Pangu Ultra is pre-trained on 13.2 trillion tokens through three data phases, extends context from 4K to 128K, then uses supervised fine-tuning and reinforcement learning.The phases target knowledge and linguistics, reasoning ability, and active learning to reason, respectively.Source: §1 · p. 2
  • The authors use 8,192 Ascend NPUs and system optimizations, including virtual-pipeline scheduling and a fused attention operator, to address pipeline bubbles and long-sequence costs.Virtual-pipeline scheduling reduces the reported pipeline-bubble ratio from 30.45% to 6.8%.Source: §1 · p. 2
  • On public benchmarks, Pangu Ultra outperforms Llama 405B and Mistral Large 2 123B on almost all major language tasks and is competitive with sparse models above 500 billion parameters.The authors also identify Ascend NPUs as suitable for exploring large-scale dense-model capabilities.Source: §1 · p. 2

2 Model Architecture

Pangu Ultra combines a 135-billion-parameter, 94-layer architecture with depth-scaled sandwich normalization, TinyInit, and a domain-aware tokenizer. The normalization and initialization choices address training stability, while the tokenizer balances domain representation and compression efficiency.

2 Model Architecture

  • Pangu Ultra has 135 billion parameters and 94 layers, uses a SwiGLU FFN, and applies Group Query Attention with 96 query heads and 8 KV heads to reduce KV-cache size.Its hidden dimension is 12,288 and FFN intermediate size is 28,672.Source: §2 · p. 2

2.1 Depth-Scaled Sandwich-Norm

  • The paper notes that increasing depth makes training stability harder to maintain, while sandwich norm alone may not prevent instability from accumulated residual-output norms.The depth-scaled initialization scheme is introduced to address this remaining concern.Source: §2.1 · pp. 2–3
  • Depth-scaled sandwich norm normalizes sub-layer outputs and initializes trainable gamma parameters in proportion to inverse network depth to help regulate layer-wise output magnitudes.It addresses instability from sub-layer output-scale fluctuations and residual accumulation; Figure 1 contrasts its normalization placement with Pre-LN.Source: §2.1 · p. 3, Figure 1 · p. 3

2.2 Model Initialization

  • TinyInit scales initialization according to model width and depth; the authors report faster loss convergence and improved downstream-task performance.The authors hypothesize that it produces more consistent parameter scales across the model, potentially facilitating optimization and convergence.Source: §2.2 · p. 3
  • The reported embedding initialization standard deviation of 0.5 achieves good model performance in the authors’ experiments.The paper notes that embedding layers may require a different initialization strategy from other layers.Source: §2.2 · pp. 3–4

2.3 Tokenizer

  • Pangu Ultra builds a unified vocabulary of 153,376 unique tokens by merging and deduplicating domain-specific vocabularies to balance domain representation and compression efficiency.The frequency analyses cover general Chinese, general English, code, and mathematics.Source: §2.3 · p. 4, Table 1 · p. 4

3 Model Training

The training pipeline has three stages—pre-training, long-context extension, and post-training—each with specific training strategies and data-construction methods intended to gradually enhance model capabilities.

3 Model Training

  • The pipeline comprises pre-training, long-context extension, and post-training, with stage-specific training strategies and data construction methods.The authors describe it as similar to training approaches used for DeepSeek-V3 and Llama 3 [22].Source: §3 · p. 4

3.1 Pre-training Stage

Pangu Ultra’s pre-training uses a diverse 13.2T-token corpus organized into general, reasoning, and annealing phases, with data quality assessment and staged training parameters. An ablation found that low-scoring data required more tokens to reach comparable performance.

3.1.1 Data Construction

  • The 13.2T-token corpus is divided into general training (12T tokens), reasoning (0.8T), and annealing (0.4T) phases.The general phase contains 7.4T and 4.6T-token subphases.Source: §3.1.1 · p. 4
  • The general phase draws mainly on English and Chinese sources, while the reasoning phase raises mathematical and coding data above 60% of its corpus.The annealing phase emphasizes instruction data at approximately 20% of its corpus, including refined short and long chain-of-thought responses.Source: §3.1.1 · pp. 4–5

3.1.2 Data Quality Assessment

  • Data quality assessment combines rule-based heuristics with model evaluation, and scores cleanliness, fluency, educational value, and richness to prioritize sampling.The Pangu-series evaluator was fine-tuned on manually annotated data and applied to a corpus exceeding 10T tokens.Source: §3.1.2 · p. 5
  • 1.6× more tokens were required for a 2.6B proxy model trained on low-scoring data to achieve performance comparable to one trained on high-quality, high-scoring data.The authors conclude that high data quality improves training efficiency.Source: §3.1.2 · p. 5

3.1.3 Pre-training Parameters

  • Pre-training uses AdamW with weight decay 0.1, epsilon 1 × 10−8, β1 = 0.9, β2 = 0.95, and gradient clipping norm 1.0.The sequence length is 4K through 12T tokens, then increases to 8K for the 12.0T–12.8T phase; batch size and learning rate also change by phase.Source: §3.1.3 · p. 5

3.2 Long Context Extension

Pangu Ultra extends context length progressively to 128K in two phases. It selects increased RoPE base frequencies using offline Needle In A Haystack performance at the target sequence lengths.

3.2 Long Context Extension

  • The final pre-training stages expand context length first to 32K and then to 128K.The two training phases use batch sizes of 384 and 96, respectively, with a learning rate of 7.5 × 10−6.Source: §3.2 · p. 6
  • Offline Needle In A Haystack evaluation at each target length determines the increased RoPE base frequency, selected to ensure relatively low initial loss in long-context training.The selected bases are 1.6 × 106 for 32K and 2.56 × 107 for 128K.Source: §3.2 · p. 6

3.3 Post-training Alignment

Pangu Ultra’s post-training aligns the model with human preferences through SFT and RL, using curated and quality-controlled instruction data. The strategy applies outcome-based rewards to further develop reasoning, alignment, and instruction-following abilities.

3.3.1 Post-training Data

  • Post-training data is sampled across domain and task-type dimensions, with hierarchical tagging and rule- and model-based validation to filter low-quality or ambiguous samples.Source: §3.3.1 · p. 6
  • Approximately six-sevenths of post-training data consists of moderately to highly challenging reasoning tasks, including mathematics, coding, and logic.The broader data pool spans domains and task types including question answering, classification, analysis, programming, tool use, finance, healthcare, and public services.Source: §3.3.1 · p. 6

3.3.2 Post-training Strategy

  • SFT establishes preliminary instruction-following capabilities before RL with outcome-based rewards further enhances reasoning, alignment, and instruction-following abilities.The RL framework uses task-specific hybrid rewards combining deterministic signals and model-based evaluations for mathematics, coding, and general problem-solving.Source: §3.3.2 · p. 6

4 Training System

Pangu Ultra’s 8,192-NPU training system combines parallelism strategies with kernel, communication, and long-context optimizations. Together, these system enhancements raise training efficiency beyond the parallelism baseline.

4.1 Computing Setup

  • The cluster contains 8,192 NPUs, with eight devices per node, full-mesh HCCS connections within nodes, and 200 Gbps RoCE inter-node links.Each NPU has 64GB of memory.Source: §4.1 · p. 7

4.2 Parallelism Strategies for Model Scaling

  • Interleaved pipeline scheduling with six virtual pipeline stages per device reduces the pipeline bubble ratio from 30.45% to 6.8%.The setup combines 128-way DP with ZERO, 8-way TP, and 8-way PP; load balancing across pipeline and virtual stages yields approximately 43% baseline MFU.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 System Optimization

  • Over 52% MFU is achieved after system optimizations, a 9% relative improvement over the 43% MFU baseline.The overall training-system result is reported for Pangu Ultra’s 135B parameters and 13.2T tokens on 8,192 Ascend NPUs.Source: §4 · p. 7, §4.3 · p. 8

4.3.1 Kernel Fusion

  • MC2 fuses matrix multiplication with All-Gather and Reduce-Scatter communication, splitting work into pipelined subtasks to overlap communication and computation.Other fusions cover attention, RMSNorm, SwiGLU, RoPE, gradient accumulation, and pipeline send/receive operations.Source: §4.3.1 · pp. 8–9, Figure 3 · p. 8

4.3.2 Optimization for Long Context Training

  • For 128K-context training, each rank computes two chunks within each subsequence to balance context-parallel workloads under reset attention masks.The system also generates masks with NPU operators and shares masks across virtual pipeline stages to avoid repeated computation and memory cost.Source: §4.3.2 · p. 9

5 Results

Pangu Ultra performs strongly across pre-training and post-training evaluations, with stable loss throughout pre-training. Ablations associate depth-scaled sandwich-norm and TinyInit with improved training behavior, while layer statistics show stable activation scales and depth-dependent normalization patterns.

5.1 Pre-Training Training Loss Curve

  • No loss spikes occur throughout pre-training; loss descends across stages, and performance metrics keep improving during the slower second interval.The second interval’s descent rate moderates under a constant learning rate.Source: §5.1 · p. 10, Figure 6 · p. 10

5.2 Pre-Training Stage

  • Pangu Ultra reaches state-of-the-art performance on most general English and all Chinese benchmarks, while remaining competitive with DeepSeek V3 on code and math tasks.It surpasses Qwen 2.5 72B and DeepSeek V3 on Chinese benchmarks and scores better than Llama 3.1 405B on most challenging benchmarks using about 29% of Llama 405B’s training FLOPs.Source: §5.2 · p. 11, Table 3 · p. 11

5.3 Post-Training and Reasoning Capability

  • Pangu Ultra achieves state-of-the-art performance on AIME 2024, MATH-500, GPQA, and LiveCodeBench while maintaining strong general-language comprehension.It shows particularly significant reasoning-task advantages over dense LLMs Qwen and Mistral-Large 2.Source: Table 4 · p. 12, §5.3 · p. 12

5.4.1 Depth-scaled Sandwich-norm

  • Depth-scaled sandwich-norm stabilizes gradient norms, eliminates loss spikes, and supports faster convergence and better benchmark performance than Pre-LN.In a 94-layer, 1.6B-parameter proxy, plain sandwich-norm still shows loss spikes, while depth-scaled sandwich-norm trains smoothly with lower loss.Source: §5.4.1 · pp. 12–13, Figure 7 · p. 13, Figure 8 · p. 13

5.4.2 Tiny Initialization

  • After 102 billion training tokens, the TinyInit model performs significantly better than the traditional-initialization baseline.The passage reports standard-deviation settings for the compared initializations, but their notation is unclear.Source: §5.4.2 · p. 13, Table 6 · p. 14

5.4.3 Layer Statistics of Pangu Ultra

  • Activation standard deviations stay consistent across layers during pre-training, while top-1 activation values progressively decrease with depth; post-norm γ standard deviation rises with depth, whereas pre-norm γ standard deviation remains relatively constant.Source: §5.4.3 · p. 14

6 Conclusion

Pangu Ultra is a 135-billion-parameter dense foundation model trained on Ascend NPUs. The authors report stable training and competitive reasoning performance against larger sparse models, demonstrating Ascend’s capacity to train dense models at this scale.

6 Conclusion

  • Pangu Ultra is a 135-billion-parameter dense language foundation model trained on Ascend NPUs.The authors report pre-training on 13.2 trillion tokens and long-context extension on 8,192 Ascend NPUs.Source: §6 · p. 14
  • Depth-scaled sandwich-norm enables stable training without significant loss spikes; supervised fine-tuning and reinforcement learning further enhance reasoning capabilities.Source: §6 · pp. 14–15
  • Pangu Ultra surpasses leading dense models such as Llama 405B and Mistral Large 2 and performs competitively against larger sparse models such as DeepSeek-R1.The authors state that Ascend NPUs can train dense models with hundreds of billions of parameters.Source: §6 · p. 15

Related papers

Loading 2504.07866v2…