Source-linked AI summary
PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, Chen Li, Ziyan Gong, Yifan Yao, Xinjing Huang, Jun Wang, Jianfeng Yu, Qi Guo, Yue Yu, Yan Zhang, Jin Wang, Hengtao Tao, Dasen Yan, Zexuan Yi, Fang Peng, Fangqing Jiang, Han Zhang, Lingfeng Deng, Yehong Zhang, Zhe Lin, Chao Zhang, Shaojie Zhang, Mingyue Guo, Shanzhi Gu, Gaojun Fan, Yaowei Wang, Xuefeng Jin, Qun Liu, Yonghong Tian
TL;DR
The paper addresses the limited availability of very large Chinese pretrained language models and the difficulty of scaling their training. It develops PanGu-α with a Chinese corpus, query-layer architecture, and MindSpore Auto-parallel training, finding generally improved performance with growing model capacity across diverse tasks. Large-scale PLMs still incur expensive inference and greater software and hardware challenges at trillion-parameter scales.
Problem
Very large Chinese pretrained language models are less publicly accessible, while training PLMs beyond 10 billion parameters presents substantial scaling challenges.
Method
PanGu-α uses a Transformer-based autoregressive architecture with an added query layer, pretrained on a 1.1TB Chinese corpus and trained using MindSpore Auto-parallel.
Results
Across diverse Chinese NLP tasks and generation scenarios, performance generally improves as model capacity grows, with PanGu-α 2.6B outperforming CPM 2.6B on most evaluated zero-, one-, and few-shot tasks.
Takeaways & Limitations
PanGu-α demonstrates broad few-shot and zero-shot capabilities for Chinese language understanding and generation at scales up to 200 billion parameters.
Takeaways & Limitations
Inference for super-large language models remains expensive, while scaling to trillions of parameters creates further software and hardware challenges.
Abstract
from arXiv · showhide
Large-scale Pretrained Language Models (PLMs) have become the new paradigm for Natural Language Processing (NLP). PLMs with hundreds of billions parameters such as GPT-3 have demonstrated strong performances on natural language understanding and generation with \textit{few-shot in-context} learning. In this work, we present our practice on training large-scale autoregressive language models named PanGu-$α$, with up to 200 billion parameters. PanGu-$α$ is developed under the MindSpore and trained on a cluster of 2048 Ascend 910 AI processors. The training parallelism strategy is implemented based on MindSpore Auto-parallel, which composes five parallelism dimensions to scale the training task to 2048 processors efficiently, including data parallelism, op-level model parallelism, pipeline model parallelism, optimizer model parallelism and rematerialization. To enhance the generalization ability of PanGu-$α$, we collect 1.1TB high-quality Chinese data from a wide range of domains to pretrain the model. We empirically test the generation ability of PanGu-$α$ in various scenarios including text summarization, question answering, dialogue generation, etc. Moreover, we investigate the effect of model scales on the few-shot performances across a broad range of Chinese NLP tasks. The experimental results demonstrate the superior capabilities of PanGu-$α$ in performing various tasks under few-shot or zero-shot settings.
1 Introduction
PanGu-α addresses limited access to large Chinese pretrained models by training an autoregressive model up to 200 billion parameters. Its experiments examine scaling, generation, and few-shot performance across diverse Chinese NLP tasks, while MindSpore Auto-parallel supports large-scale training.
- PanGu-α is proposed as a Chinese pretrained language model with up to 200 billion parameters, addressing GPT-3’s limited access and primarily English training data.
- Training PLMs beyond 10 billion parameters introduces challenges in model design, including slow convergence or divergence as model size increases.
- The study trains PanGu-α models with 2.6B, 13B, and 200B parameters on a 1.1TB Chinese corpus, then evaluates language modeling and text generation.
- Evaluations cover dialogue generation, summarization, question answering, cloze tasks, reading comprehension, reasoning, inference, and classification.
- Across the evaluated tasks, performance generally improves as model capacity grows, including under task-agnostic few-shot settings.
- MindSpore Auto-parallel is open-sourced to facilitate large-scale pretraining with minimal or zero code modifications from standalone training.
2 Model
PanGu-α is a Transformer-based autoregressive language model that predicts each next token from preceding context, with an added query layer to induce the expected output. Its design uses multi-head attention, feed-forward layers, pre-layer normalization, and scales across 2.6B, 13B, and 200B parameter configurations.
- Overview: PanGu-α models token generation autoregressively, maximizing the log-likelihood of each token conditioned on its preceding context.The sequence contains N tokens, and the probability of each token is conditioned on x1:n−1 and model parameters θ.
- Transformer Layers: The architecture uses a uni-directional Transformer decoder with multi-head attention and fully connected feed-forward sub-layers.The Transformer layer applies multi-head attention followed by a feed-forward network.
- Transformer Layers: Pre-layer normalization is applied to both multi-head attention and feed-forward sub-layers to make Transformer training easier and faster.
- Query Layer: An additional query layer sits above the Transformer stack and uses a next-position embedding as its attention query to predict the next token.Its subsequent multi-head attention and feed-forward computations follow the original Transformer, producing the final output for token prediction.
- Model Configurations: PanGu-α is evaluated in 2.6B, 13B, and 200B parameter configurations with corresponding model dimensions and attention-head settings.Table 1 reports the model sizes and hyperparameters for these three configurations.
3 Dataset
The paper constructs a 1.1TB high-quality Chinese corpus from nearly 80TB of heterogeneous raw data through iterative cleaning, filtering, evaluation, and deduplication. Separate 100GB and 1TB datasets support the medium and 200B-parameter models.
- Existing Chinese web data varies greatly in quality, while preprocessing requires substantial storage and computation and lacks well-defined quality metrics.
- 1.1TB of high-quality Chinese text is constructed from nearly 80TB of public datasets, web pages, encyclopedias, news, and e-books.
- The pipeline combines rule-based cleaning, model-based filtering, and cross-source text deduplication, with manual and model-based evaluations guiding iterative refinement.
- 724 sensitive words and a spam classifier trained with 10K manually selected junk documents are used to remove harmful, advertising, and spam content.
- A distributed deduplication algorithm processes 500GB of documents in 3.5 hours, whereas Spark MinHashLSH takes more than 8 hours for less than 200MB.
- The final corpus yields 100GB and 1TB training datasets for the 2.6B/13B and 200B models, respectively, with source sampling proportions based on evaluated quality.
4 System
Training PanGu-α 200B requires distributed execution because its memory demands exceed individual AI processors. MindSpore Auto-parallel combines multiple parallelism dimensions and topology-aware scheduling to balance memory, communication, and throughput across the cluster.
- Storing PanGu-α 200B requires around 750 GB, while individual modern AI processors provide only about 30–40 GB of memory.
- Auto-parallel addresses distributed training through five-dimensional parallelism and topology-aware scheduling while separating parallel implementation from model definition.
- Data parallelism partitions batches and synchronizes gradients, whereas op-level model parallelism slices operator tensors to reduce parameter and activation memory.
- Each parallelism dimension trades computation or communication overhead for memory or throughput benefits, requiring a balanced composition point.
- Pipeline parallelism is mapped across servers according to their hierarchical bandwidths, exploiting higher communication bandwidth within servers than across racks.
- For PanGu-α 200B, 64 layers are divided into 16 pipeline stages with 4 layers each, while operator parameters and tensors are further partitioned.
- Auto-parallel can infer required communication and tensor redistribution operators and enables parallel training by adding annotations to standalone model code.
5 Experiments
The experiments evaluate PanGu-α across training behavior, broad Chinese NLP tasks, few-shot learning, and open-ended generation. Larger models generally improve language modeling and few-shot performance, while PanGu-α 2.6B and 13B also compare favorably with baselines in several task groups.
- 5.1 Training Details: 2048 Ascend 910 AI processors train the PanGu-α models under MindSpore, with the 200B model using 2048 processors initially and 1024 later.The models use BPE tokenization with a 40,000-word vocabulary and sequence length 1024.
- 5.1 Training Details: Training loss converges to 2.49 for 200B, 2.58 for 13B, and 2.64 for 2.6B, while all curves remain decreasing at training end.The authors interpret the continued decrease as evidence that the models are under-trained and may improve with further training.
- 5.2 Task Description: The evaluation covers zero-shot, one-shot, and few-shot settings without finetuning across seven categories of Chinese NLP tasks.The categories include cloze and completion, reading comprehension, closed-book QA, Winograd-style tasks, commonsense reasoning, NLI, and text classification.
- 5.3 Generation Method: Classification tasks are evaluated through perplexity comparison, while generation tasks are solved by prompting autoregressive answer generation.Task-specific templates and decoding strategies are described for the evaluated tasks.
- 5.4 Results: PanGu-α 2.6B surpasses CPM 2.6B on more than 11 zero-shot, 12 one-shot, and 14 few-shot tasks, with a 6-point average gain on generation tasks.It is comparable on CMNLI and OCNLI but slightly worse on TNEWS and IFLYTEK.
- 5.4 Results: PanGu-α 13B outperforms 2.6B on all generation tasks and most perplexity tasks, with a 3-point average improvement and more than 10-point few-shot gains on CMRC2018, DRCD, and WebQA.The 13B model performs worse on NLI tasks than the 2.6B model.
- 5.5 Natural Language Generation Examples: Roughly 90% of generated PanGu-α 200B examples are human-acceptable in poetry and duilian, summarization, dialogue, and fiction generation.The examples receive no post-editing except truncation when generation fails to stop at a reasonable point.
6 Conclusion
The paper presents PanGu-α, large-scale Chinese autoregressive language models reaching 200 billion parameters, and identifies open problems for scaling and applying large PLMs. These include efficient few-shot use, cheaper inference, trillion-parameter training, and multimodal pretraining.
- 6 Conclusion: PanGu-α comprises Chinese autoregressive language models with up to 200 billion parameters, developed under MindSpore and trained on 2048 Ascend AI processors.
- 6 Conclusion: Efficient few-shot algorithms and appropriate use of large PLMs remain open questions because their behaviors are not yet systematically studied.
- 6 Conclusion: Inference cost remains expensive, motivating research on model compression and acceleration without sacrificing much performance.
- 6 Conclusion: Trillion-parameter training presents software and hardware challenges, while MoE and Switch Transformers may help reduce training and inference costs.
- 6 Conclusion: Pretrained multimodal models integrating language, vision, and speech data are another proposed direction for future study.