PaperlayerView exact revision on arXiv

Source-linked AI summary

CPM: A Large-scale Generative Chinese Pre-trained Language Model

Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan, Fanchao Qi, Xiaozhi Wang, Yanan Zheng, Guoyang Zeng, Huanqi Cao, Shengqi Chen, Daixuan Li, Zhenbo Sun, Zhiyuan Liu, Minlie Huang, Wentao Han, Jie Tang, Juanzi Li, Xiaoyan Zhu, Maosong Sun

arXiv:2012.00413v1cs.CL

TL;DR

GPT-3’s primarily English training corpus and unavailable parameters make it challenging to apply to Chinese NLP. The report presents CPM, a large-scale Chinese generative pre-trained language model, and reports strong performance across many tasks in few-shot and zero-shot settings.

  • Problem

    GPT-3’s primarily English training corpus and unavailable parameters make it challenging to apply to Chinese NLP, while prior Chinese pre-trained models are limited by model size.

    Source: §1 · p. 1

  • Method

    CPM is a Chinese autoregressive language model trained with generative pre-training on large-scale Chinese corpora.

    Source: §1 · pp. 1–2

  • Results

    CPM achieves strong performance on many NLP tasks in few-shot and zero-shot settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The report presents CPM as a large Chinese pre-trained model for tasks including conversation, essay generation, cloze tests, and language understanding.

    Source: §1 · p. 2

    CPM does not achieve very high scores on either question-answering benchmark, which the authors guess may relate to the pretraining-data format.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

Pre-trained Language Models (PLMs) have proven to be beneficial for various downstream NLP tasks. Recently, GPT-3, with 175 billion parameters and 570GB training data, drew a lot of attention due to the capacity of few-shot (even zero-shot) learning. However, applying GPT-3 to address Chinese NLP tasks is still challenging, as the training corpus of GPT-3 is primarily English, and the parameters are not publicly available. In this technical report, we release the Chinese Pre-trained Language Model (CPM) with generative pre-training on large-scale Chinese training data. To the best of our knowledge, CPM, with 2.6 billion parameters and 100GB Chinese training data, is the largest Chinese pre-trained language model, which could facilitate several downstream Chinese NLP tasks, such as conversation, essay generation, cloze test, and language understanding. Extensive experiments demonstrate that CPM achieves strong performance on many NLP tasks in the settings of few-shot (even zero-shot) learning. The code and parameters are available at https://github.com/TsinghuaAI/CPM-Generate.

1 Introduction

GPT-3’s predominantly English training data and unavailable parameters make it challenging to apply to Chinese NLP, while existing Chinese models are limited by size. CPM is a large Chinese autoregressive model that achieves strong performance across many tasks in few-shot and zero-shot settings.

1 Introduction

  • GPT-3’s predominantly English corpus and unavailable parameters complicate its use for Chinese NLP, while prior Chinese models are limited by size.The report motivates further exploration of large-scale Chinese language-model pre-training, including Chinese vocabulary construction and training strategy.Source: §1 · p. 1
  • CPM is a 2.6-billion-parameter Chinese autoregressive language model trained on 100GB of Chinese data.The report describes it as, to the authors’ knowledge, the largest Chinese pre-trained language model.Source: §1 · p. 2
  • The report’s contributions include a new sub-word vocabulary adapted to Chinese corpora and a batch size of 3,072 for more stable training.The vocabulary is based on a word-segmented corpus.Source: §1 · p. 2
  • CPM achieves strong performance on many Chinese NLP tasks in few-shot and zero-shot settings.The report lists conversation, essay generation, cloze tests, and language understanding as potential downstream tasks.Source: §1 · p. 2

2 Our Approach

CPM uses a left-to-right Transformer decoder and adapts its vocabulary and training strategy for Chinese. The approach also describes its pretraining data and the settings used to train the models.

2.1 Chinese PLM

  • CPM uses a left-to-right Transformer decoder and a Chinese sub-word vocabulary containing both words and characters.The vocabulary is built from a word-segmented corpus; the approach also uses a large batch size to stabilize training.Source: §2.1 · p. 2

2.2 Data Processing

  • Pretraining combines encyclopedia, news, novels, and Q&A texts, concatenating documents with an end-of-document token to use the input length.The training-data table is identified as giving further details, but its values are not supplied here.Source: §2.2 · p. 2, Table 2 · p. 2

2.3 Pre-training Details

  • Pretraining uses a learning rate of 1.5 × 10^-4, batch size 3,072, sequence length 1,024, and 20,000 steps, including 5,000 warm-up steps.The largest model is partitioned across GPUs along the width dimension during training; the first version uses dense attention.Source: §2.3 · pp. 2–3, §2.1 · p. 2

3 Experiments

The experiments evaluate CPM on text classification, idiom cloze, dialogue, question answering, and entity generation. Results show benefits associated with model scale and few-shot prompting, alongside task-specific performance limits.

3.1 Text Classification

  • CPM-Large significantly outperforms small and medium on TNEWS, IFLYTEK, and OCNLI, while small and medium are close on TNEWS and OCNLI.These are zero-shot evaluations using validation sets without training instances.Source: §3.1 · p. 3, Table 3 · p. 3
  • On OCNLI, small and medium are close to neutral-only accuracy (0.374), suggesting zero-shot natural language inference is harder than the other classification tasks.The OCNLI validation set is unbalanced, and neutral-only prediction yields about 0.374 accuracy.Source: §3.1 · p. 3

3.3 Dialogue Generation

  • CPM outperforms CDial-GPT by a large margin few-shot and remains better supervised, especially on diversity; larger CPMs produce more diverse responses with reasonable embedding metrics.Dialogue quality is assessed with embedding-based similarity and distinct n-gram metrics.Source: Table 5 · p. 4, §3.3 · p. 5, Table 7 · p. 5

3.2 Chinese Idiom Cloze

  • Unsupervised CPM-Large outperforms supervised CPM-Small and is comparable to supervised CPM-Medium; it ranks candidate-filled passages by perplexity.For each blank, the model scores ten completed passages and selects the one with lowest perplexity.Source: Table 4 · p. 3, §3.2 · p. 4

3.4 Question Answering

  • Across three datasets, larger CPMs perform better and one-shot beats zero-shot, but generated answers are often long and repetitive, contributing to low scores.The authors suggest the low benchmark scores may relate to the pre-training data format.Source: Table 8 · p. 5, §3.4 · p. 5

3.5 Entity Generation

  • CPM-Large leads the three model sizes, and two-triple prompts yield results comparable to four-triple prompts in extreme few-shot entity generation.The authors suggest CPM can imitate the format and probe factual knowledge to generate a tail entity.Source: Table 9 · p. 6, §3.5 · p. 6

4 Future Work

Future work proposes expanding CPM’s scale and data, improving pretraining efficiency, and broadening its training inputs.

4 Future Work

  • Future plans include adding more training data and increasing model size to explore large-scale Chinese pre-trained models.The authors also plan to add diverse data.Source: §4 · p. 6
  • Because pretraining is extremely expensive, the authors plan to optimize the training framework, including inter-node data transfer, to accelerate it.Source: §4 · p. 6
  • The authors plan to reduce model size through model compression.They cite LAMB, DeepSpeed, and model-compression approaches as related work.Source: §4 · p. 6
  • Planned data extensions include multilingual text and joint learning from text and knowledge graphs.The proposed Chinese-centered multilingual model and joint text–knowledge-graph model are intended to enhance performance and general intelligence.Source: §4 · p. 6

Disclaimer of Warranties

The disclaimer limits CPM-generated text to technical and scientific use and disclaims endorsement of its content.

Disclaimer of Warranties

  • CPM-generated text comes from a neural network trained on a large number of texts.Source: Disclaimer of Warranties · p. 7
  • Generated text does not represent the authors’ official attitudes or preferences.Source: Disclaimer of Warranties · p. 7
  • The authors restrict intended use to technical and scientific purposes and ask readers not to propagate rights-infringing or socially immoral text.They ask affected readers to contact them so they can address the issue.Source: Disclaimer of Warranties · p. 7

A Contributions

The contribution acknowledgments assign work across model implementation, research leadership, infrastructure, data, downstream tasks, demonstrations, and evaluation.

A Contributions

  • Zhengyan Zhang, Xu Han, and Hao Zhou implemented the large-scale models and model-parallel strategies.Source: A Contributions · p. 8
  • Huanqi Cao, Shengqi Chen, Daixuan Li, and Zhenbo Sun built the training infrastructure, while Pei Ke, Deming Ye, Jian Guan, Fanchao Qi, and Xiaozhi Wang collected, filtered, and deduplicated training data.Source: A Contributions · p. 8
  • Zhengyan Zhang, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, and Haozhe Ji implemented downstream tasks and the supporting software framework.Source: A Contributions · p. 9
  • Hao Zhou, Guoyang Zeng, Xu Han, and Yanan Zheng implemented language-generation and knowledge-retrieval demos using CPM; Guoyang Zeng conducted human evaluations.Source: A Contributions · p. 9
  • Zhiyuan Liu, Minlie Huang, and Wentao Han designed and led the research; Jie Tang, Juanzi Li, Xiaoyan Zhu, and Maosong Sun provided advice.Source: A Contributions · p. 9

Related papers

Loading 2012.00413v1…