Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence

1 Introduction

• Existing foundation models are dominated by English, while the Chinese-language community lacks systematic research resources.Source: §1 · p. 1
• The ecosystem combines Fengshenbang Model, Fengshen Framework, and Fengshenbang Benchmark with pre-trained models, fine-tuning applications, benchmarks, and datasets.Source: §1 · pp. 1–2
• The model offering spans general- and special-purpose NLP, with associated downstream fine-tuning models for users with limited resources.Source: §1 · p. 2
• The framework provides standardized data processors, tutorials, a Docker-like environment, and APIs for low-cost utilization and model adaptation.Source: §1 · p. 2
• The benchmark module supports fair comparisons and community progress tracking.Source: §1 · p. 2

2.1.1 User-centered Taxonomy (UCT)

• UCT organizes general-purpose NLU, NLG, and NLT alongside special-purpose demands; NLT specifically denotes source-to-target transformation tasks.Source: §2.1.1 · pp. 2–3
• The taxonomy includes multimodal scenarios such as text-to-image generation, image captioning, cross-modal retrieval, visual question answering, and speech tasks.Source: §2.1.1 · p. 3
• For domain-specific models, continuous pre-training avoids training from scratch and reduces computational resource consumption.Source: §2.1.1 · p. 3
• The project also plans to develop experimental NLP models with technology companies and universities.Source: §2.1.1 · p. 3

2.1.2 Model Selection

• Model selection prioritizes powerful performance, diversity across tasks and architectures, and practical usability, including out-of-the-box use.Source: §2.1.2 · pp. 3–4

2.1.3 Naming Convention

• The naming convention uses Series for a task category, Model for architecture, Parameter for parameter count, and Extra for setting details.Source: §2.1.3 · p. 4

2.2.1 MegatronBERT

• Erlangshen-MegatronBert-1.3B topped FewCLUE and ZeroCLUE and the CLUE semantic-matching task, with additional first-place task rankings.The authors also report human-level performance in CHID and TNEWS.Source: §2.2.1 · p. 4

2.2.2 ZEN

• ZEN1 uses n-gram methods to learn textual granularity and gains performance when trained on one small corpus; ZEN2 pre-trains n-gram-enhanced encoders on large-scale data.The project releases ZEN1 and ZEN2 in collaboration with the ZEN team.Source: §2.2.2 · p. 5

2.2.3 RoBERTa

• Chinese RoBERTa follows RoBERTa, using whole-word masking in masked language modeling and pretraining on 180 GB of WuDao Corpora.The listed models include 110M and 330M variants for NLI, sentiment, and similarity.Source: §2.2.3 · p. 5

2.2.4 DeBERTa

• Chinese DeBERTa-v2 models use whole-word masking in masked language modeling and pretrain on 180 GB of WuDao Corpora.The listed versions include 186M, 320M, and 710M models, plus two 97M variants.Source: §2.2.4 · p. 5

2.2.5 Longformer

• Chinese Longformer uses whole-word masking, 180 GB of WuDao Corpora, and Rotary Position Embedding to address uneven sequence lengths in the pretraining corpus.The listed models have 110M and 330M parameters.Source: §2.2.5 · p. 5

2.2.6 UBERT

• UBERT is a BERT-based multi-task framework that won the 2022 AIWIN Chinese insurance small-sample multi-task challenge and is released with over 70 datasets.The authors describe use in NLI, entity recognition, and reading comprehension.Source: §2.2.6 · p. 5

2.2.7 UnifiedMC (temporary)

• UnifiedMC explores encoder-based pretrained models for zero-shot and few-shot use, and its models topped FewCLUE and ZeroCLUE on August 30, 2022.The authors say code and pretraining-task details would be made public upon acceptance.Source: §2.2.7 · p. 5

2.3 Wenzhong (NLG)

• Wenzhong focuses on NLG and includes Chinese GPT-2 models, including a 3.5B model pretrained on CLURCorpus2020 and a 110M model pretrained on the 300G Wudao Corpus.Wenzhong2.0-GPT2-3.5B-chinese is described as sharing the structure of Wenzhong-GPT2-3.5B and using the 300G Wudao Corpus.Source: §2.3 · p. 5, §2.3.1 · pp. 5–6

2.4 Randeng (NLT)

• Randeng targets source-to-target transformations such as summarization, with BART, MegatronT5, PEGASUS, and T5 variants.The BART model uses a BERT tokenizer and WuDao Corpora; MegatronT5 uses Megatron-LM and WuDao Corpora.Source: §2.4 · p. 6, §2.4.1 · p. 6, §2.4.2 · p. 6

2.4.1 BART

• Randeng-BART uses a BERT tokenizer and 180 GB of WuDao Corpora, with one model applying that tokenizer because it usually performs better for Chinese tasks.The listed models include 139M and 759M variants.Source: §2.4.1 · p. 6

2.4.2 MegatronT5

• Randeng-MegatronT5 applies the Megatron-LM method to large-scale T5 pretraining on 180 GB of WuDao Corpora.The listed model is Randeng-MegatronT5-770M.Source: §2.4.2 · p. 6

2.4.3 PEGASUS

• Randeng-PEGASUS follows PEGASUS guidelines for Chinese summarization, using 180 GB of WuDao Corpora and jieba plus a BERT tokenizer.The section also describes Chinese mT5 training that retrains Chinese and English vocabulary and embeddings, then applies corpus-adaptive pretraining on WuDao Corpora.Source: §2.4.3 · p. 6, §2.4.4 · p. 6

2.4.5 Transformer-Denoise

• Transformer-Denoise pretrains Transformer-XL on the 180G Wudo corpus, then fine-tunes on a developed dataset to reconstruct fluent text from inputs corrupted by insertions, swaps, deletions, replacements, or sentence reordering.The listed model is Randeng-Transformer-1.1B-Denoise.Source: §2.4.5 · p. 6

2.5 Taiyi (MM)

• Taiyi targets cross-modal scenarios, including work on protein-structure prediction and speech-text representations.Taiyi-CLIP is applied to text-to-image generation, including generating images from Chinese ancient poems.Source: §2.5 · p. 6, §2.5.1 · p. 6

2.5.1 CLIP

• Taiyi-CLIP uses Chinese RoBERTa as its language encoder and CLIP's ViT as its vision encoder, freezing the vision encoder while tuning the language encoder.It is pretrained on Noah-Wukong and Zero-Corpus; the authors describe it as the only open-sourced Chinese CLIP in the Hugging Face community at the time.Source: §2.5.1 · p. 6

2.5.2 MAP (temporary)

• MAP uses a module for multimodal uncertainty and specialized pretraining strategies; fine-tuned models achieve state-of-the-art performance on challenging downstream tasks and help unimodal tasks.The authors describe the model as exploratory and say code and pretraining details would be made public upon acceptance.Source: §2.5.2 · pp. 6–7

2.6 Yuyuan (Domain)

• Yuyuan addresses the need for high-quality domain-specific pretrained language models, noting that FinBERT and general BERT share 41% of their vocabulary tokens.The passage links missing domain-specific vocabulary terms to incorrect word representations and impaired model learning.Source: §2.6 · p. 7

2.6.1 BioBART

• BioBART applies BART-large to about 41 GB of PubMed abstracts and shows large improvements across benchmarks, with competitive or superior results to existing state-of-the-art methods.The evaluated language-generation tasks include dialogue and summarization, alongside entity linking and named entity recognition.Source: §2.6.1 · p. 7

2.6.2 GPT2

• Yuyuan-GPT2-3.5B is pretrained on 50 GB of PubMed data and can judge facts using perplexity; its QA variant converts interrogative phrases to declarative ones.The authors describe it as the largest open-source GPT-2 model in the medical domain.Source: §2.6.2 · p. 7

2.7 TBD (Exploration)

• The section identifies Zhouwenwang models as outcomes of joint efforts with Zhuiyi Technology.The listed models are Zhouwenwang-Unified-1.3B and Zhouwenwang-Unified-110M.Source: §2.7 · p. 7

3 Fengshen Framework

• Fengshen Framework integrates Hugging Face, Megatron-LM, PyTorch-Lightning, and DeepSpeed, supporting pretraining models over 10 billion parameters on terabyte-level data and downstream fine-tuning.It provides distributed training and memory-saving approaches, plus code and examples for open-source models and applications.Source: §3 · p. 7
• The framework lists 300% training acceleration versus original Torch, CPU/GPU/TPU support, and distributed methods such as DDP and Zero Optimizer without code modifications.It also supports terabyte-level datasets and provides pretraining and downstream examples.Source: §3 · p. 7

3.1 Architecture

• The described application flow encapsulates data processing, encapsulates the model structure, and configures plugins.The paper says it provides usage details for distributed training, fine-tuning, and large-scale model applications.Source: §3.1 · p. 8, §3.2 · p. 8

3.3 Case Study

• Only a few lines are required to construct models in Fengshen Framework, and users of frameworks such as Hugging Face face almost no cost to get started.The paper points to Figure 3 for this case study.Source: §3.3 · p. 8

4 Fengshenbang Benchmark

• The benchmark combines different types of Chinese natural-language tasks and plans to release a general pre-training dataset.The project presents benchmarks as a basis for evaluating models and supporting fair comparison.Source: §4 · p. 8

4.1 Task Criteria

• Benchmark criteria include widely used datasets, more challenging future-oriented evaluation, and real-world applicability through industry collaboration.The authors propose carefully translating and checking popular datasets and collecting real-world data with companies.Source: §4.1 · p. 8

4.2.1 Chinese-SuperGLUE

• Chinese-SuperGLUE is proposed to evaluate Chinese models because SuperGLUE is widely adopted but available only in English.The authors say the related paper and leaderboard are forthcoming.Source: §4.2.1 · p. 8

4.2.2 QAKM

• QAKM evaluates models’ knowledge by asking them to learn domain-specific knowledge and answer unseen questions from a dataset.The task is included in NLPCC.Source: §4.2.2 · pp. 8–9

5 Summary and Future Work

• Fengshenbang’s Model, Framework, and Benchmark sub-projects support different aspects of Chinese intelligent systems’ progress.The project is described as ongoing, with sub-projects to be continually updated and refined with community contributions.Source: §5 · p. 9

Ethical Considerations

• The authors identify potential bias, environmental issues, harmful judgments in labeled data, and inappropriate framework use as ethical concerns.They plan a detailed study in the next report version and encourage public debate about project use.Source: Ethical Considerations · p. 9

7.1 模型设计

• The user-centered taxonomy groups general NLP needs into NLU, NLG, and NLT, alongside special needs such as multimodality and domain tasks.Model selection considers strength, diversity, and practicality, and the naming scheme identifies series, model structure, parameter count, and extra settings.Source: §7.1.1 · p. 11, §7.1.2 · p. 12, §7.1.3 · p. 12

7.2.1 MegatronBERT

• Erlangshen-MegatronBERT uses whole-word masking, knowledge-based dynamic masking, sentence-order prediction, and Pre-LN during pre-training.The model is pretrained on the 180 GB Wudao corpus; Pre-LN is used to address loss increases observed with post-LN as model size grows.Source: §7.2.1 · p. 12
• Erlangshen-MegatronBert topped FewCLUE and ZeroCLUE at the reported dates and ranked first in CLUE semantic matching.The authors also report first-place results on specified subtasks, including performance above human performance on CHIDF and TNEWS in FewCLUE.Source: §7.2.1 · pp. 12–13

7.2 二郎神(NLU)

• Chinese ZEN, RoBERTa, DeBERTa, and Longformer models adapt or extend existing architectures, with whole-word masking and Wudao pre-training used for the latter three.ZEN incorporates n-gram information; Longformer uses rotary position embeddings to address uneven sequence lengths in the pre-training corpus.Source: §7.2.2 · p. 13, §7.2.3 · p. 13, §7.2.4 · p. 13

7.2.6 UBERT

• UBERT is a unified multi-task, multi-objective extraction framework pretrained on more than 70 datasets because the competition data was no longer available.The authors report first place on both competition leaderboards and say UBERT can also be used for NLI, entity recognition, and reading comprehension.Source: §7.2.6 · p. 13

7.3.1 GPT2; 7.4 燃灯(NLT)

• The work implements Chinese GPT-2 models—3.5B pretrained on CLURCorpus2020 or 300G Wudao, plus a 110M model pretrained on 300G Wudao—and introduces Randeng for NLT source-to-target tasks such as summarization.For summarization, Randeng-PEGASUS followed PEGASUS design and used Wudao 180G for pretraining, with Jieba and BERT tokenizers. A Transformer-XL-structured denoising experiment was pretrained on Wudao 180G, fine-tuned on a constructed denoising dataset, and used to reconstruct fluent, clean text from noisy inputs.Source: §7.3.1 · p. 13, §7.4 · p. 14, §7.4.3 · p. 14

7.5 太乙(MM)

• The Taiyi series targets multimodal tasks; Taiyi-CLIP uses Chinese RoBERTa as its language encoder and ViT as its vision encoder, and is applied to text-to-image generation, including images from Chinese classical poetry.The authors state that, to their knowledge, Taiyi-CLIP was the Hugging Face community’s only open-source Chinese CLIP.Source: §7.5 · p. 14, §7.5.1 · p. 14, §7.5.2 · p. 14

7.6 余元(Domain); 7.7 系列名待定(探索); 9.1 任务选择的标准; 9.2.1 Chinese-SuperGLUE; 9.2.2 QAKM

• The section reports BioBART improvements across biomedical benchmarks, describes a framework for large-scale training, and proposes benchmark criteria and Chinese-SuperGLUE.It also identifies ethical risks from real-world deployment and says a fuller discussion is planned for a later report.Source: §7.6.1 · p. 15, §7.7 · p. 15, §9.1 · p. 16

Erlangshen models

• The listed NLU models include Erlangshen DeBERTa-v2, Longformer, MegatronBert, RoBERTa, UBERT, and ZEN variants.The appendix includes different parameter sizes and task-specific variants.Source: A A List of Fengshenbang Models · pp. 19–20

Generation models

• The list includes Randeng BART, MegatronT5, PEGASUS, T5, Transformer-Denoise, and Wenzhong GPT-2 models.Several names distinguish model size or summarization and denoising variants.Source: A A List of Fengshenbang Models · pp. 20–21

Multimodal and domain models

• Other listed offerings include Taiyi CLIP and vision models, Yuyuan biomedical models, and Zhouwenwang unified models.The appendix gives multiple named variants within these groups.Source: A A List of Fengshenbang Models · pp. 20–21

• The ongoing Fengshenbang open-source effort is maintained by engineers, researchers, and interns at the CCNL Research Center in IDEA.Source: B Author Contributions · p. 21
• Project managers initiated Fengshenbang and its sub-projects.Source: B Author Contributions · p. 21
• Contributors are listed separately for Fengshenbang Model, Fengshen Framework, and Fengshenbang Benchmark; Junjie Wang and Yuxiang Zhang are also listed for paper writing.Source: B Author Contributions · p. 21
• Lin Zhang is listed among the paper writers, with a note identifying the CTO Lab in IDEA.Source: B Author Contributions · p. 21