Source-linked AI summary
CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Hang Yan, Fei Yang, Li Zhe, Hujun Bao, Xipeng Qiu
TL;DR
Existing Chinese PTMs generally specialize in understanding or generation, motivating a model that can exploit shared knowledge across both tasks. CPT uses a shared encoder with separate MLM- and DAE-trained decoders, achieving competitive results across Chinese NLU and NLG tasks while accelerating generation.
Problem
Existing Chinese PTMs usually use different architectures and pre-training tasks for NLU and NLG, requiring task-dependent backbone selection.
Method
CPT combines a shared encoder with separate understanding and generation decoders, pre-trained jointly with MLM and DAE.
Results
CPT achieves competitive results across a wide range of Chinese NLU and NLG tasks and provides 1.4× ∼1.5× speedup over BART in base size.
Takeaways & Limitations
CPT provides a flexible Chinese pre-trained model for both language understanding and generation, with faster text-generation inference from its shallow generation decoder.
Takeaways & Limitations
The shallow generation decoder may affect long-text generation, although the reported performance gaps remain small.
Abstract
from arXiv · showhide
In this paper, we take the advantage of previous pre-trained models (PTMs) and propose a novel Chinese Pre-trained Unbalanced Transformer (CPT). Different from previous Chinese PTMs, CPT is designed to utilize the shared knowledge between natural language understanding (NLU) and natural language generation (NLG) to boost the performance. CPT consists of three parts: a shared encoder, an understanding decoder, and a generation decoder. Two specific decoders with a shared encoder are pre-trained with masked language modeling (MLM) and denoising auto-encoding (DAE) tasks, respectively. With the partially shared architecture and multi-task pre-training, CPT can (1) learn specific knowledge of both NLU or NLG tasks with two decoders and (2) be fine-tuned flexibly that fully exploits the potential of the model. Moreover, the unbalanced Transformer saves the computational and storage cost, which makes CPT competitive and greatly accelerates the inference of text generation. Experimental results on a wide range of Chinese NLU and NLG tasks show the effectiveness of CPT.
1 Introduction
CPT unifies Chinese language understanding and generation through a shared encoder and two specialized decoders, addressing the split between existing PTMs. Its multi-task pre-training, flexible fine-tuning, and unbalanced design target broad performance with faster generation inference.
- Existing Chinese PTMs typically focus on either language understanding or language generation, requiring backbone selection by downstream task.
- CPT divides a Transformer encoder-decoder into a shared encoder, an understanding decoder, and a generation decoder.The understanding decoder uses full self-attention, while the generation decoder uses masked self-attention.
- The two decoders learn task-specific knowledge through MLM and DAE, while the shared encoder learns common language representations.
- CPT supports at least five fine-tuning modes for classification, enabling mode selection according to downstream-task characteristics.
- The deeper shared encoder and shallower decoders reduce computational and storage costs, while the shallow generation decoder accelerates text-generation inference.
- Experiments across Chinese classification, sequence labeling, reading comprehension, summarization, and data-to-text generation show competitive results with state-of-the-art methods.
2 Related Work
Prior Chinese PTMs separately target understanding, generation, or full encoder-decoder modeling, while unified approaches impose architectural or task limitations. CPT combines MLM and DAE with separated decoders to support both NLU and NLG efficiently.
- UniLM and GLM unify understanding and generation with shared Transformer encoders, but their architectures restrict flexible DAE pre-training.
- PALM adds MLM to a generation-focused model, whereas CPT uses an individual MLM decoder to avoid negative effects from DAE on NLU.
- Chinese PTMs based on BERT support NLU but cannot be directly adopted for text generation.
- GPT-style Chinese PTMs support autoregressive generation, but their large parameter counts impose substantial computation and storage costs.
- Full encoder-decoder Chinese PTMs include CPM-2 and Chinese BART, but Chinese BART previously lacked a corresponding version.
- CPT uses an unbalanced Transformer with MLM and DAE and is designed for competitive NLU and NLG results with parameter efficiency.
3 Model Architecture
CPT uses a shared semantic encoder with separate understanding and generation decoders. Their combinations support task-specific fine-tuning, while shallow decoders reduce parameter overhead and generation latency.
- CPT is a full Transformer variant composed of a shared encoder, an understanding decoder, and a generation decoder.
- The shared encoder uses fully connected self-attention to capture semantic representations common to understanding and generation.
- The understanding decoder is a shallow fully self-attentive Transformer encoder that receives the shared encoder output for NLU tasks.
- The generation decoder uses masked self-attention and cross-attention over the shared encoder output for autoregressive generation.
- CPT can use the shared and understanding decoders for NLU or the shared and generation decoders for NLG, with other combinations available.
- A deeper shared encoder with two shallow decoders lowers parameter overhead and accelerates generation inference, with minor performance loss.
4 Pre-Training
CPT jointly pre-trains shared and specialized components with MLM and DAE. MLM predicts masked tokens for understanding, while DAE reconstructs documents from corrupted inputs for generation.
- CPT introduces MLM and DAE as two pre-training tasks to support both language understanding and generation.
- MLM trains the shared encoder and understanding decoder to predict randomly masked tokens using whole-word masking.
- DAE trains the shared encoder and generation decoder to reconstruct original documents from corrupted inputs.
- DAE corruption uses whole-word token infilling and random sentence permutation.
- Multi-task pre-training enables CPT to learn for both understanding and generation and to handle downstream NLU or NLG tasks.
5 Fine-Tuning
CPT supports multiple fine-tuning patterns for classification, sequence labeling, machine reading comprehension, and conditional generation. These patterns use its separate understanding and generation decoders to match downstream task requirements.
- Sequence-Level Classification: CPT can be fine-tuned in multiple patterns, including external-classifier and prompt-based modes, to exploit the model’s full potential.The paper describes three external-classifier modes and two prompt-based modes for sequence-level classification.
- Sequence-Level Classification: CPTu uses the understanding decoder’s first [CLS] representation for BERT-style sentence classification.
- Sequence-Level Classification: CPTg uses the shared encoder and generation decoder, taking the final [SEP] representation for BART-style classification.
- Sequence-Level Classification: CPTug concatenates the understanding decoder’s first output with the generation decoder’s final output.
- Prompt-Based Classification: Prompt-based classification includes MLM prediction with label-word mapping and conditional generation scored by prompt perplexity.CPTu+p averages distributions across masked positions, while CPTg+p selects the label with the highest corresponding prompt perplexity.
- Other Downstream Tasks: For sequence labeling, CPT supplies token representations to a trainable classifier, while conditional generation fine-tuning uses only the S-Enc and G-Dec modules.The sequence-labeling figure denotes text inputs and token labels; the generation figure denotes input and target sequences.
6 Experiments
CPT is evaluated across Chinese understanding and generation tasks using multiple fine-tuning modes and model configurations. It achieves competitive or superior results across classification, sequence labeling, machine reading comprehension, and text generation, while reducing generation latency.
- Experimental setup: Experiments cover classification, sequence labeling, machine reading comprehension, summarization, and data-to-text generation across Chinese datasets.The study compares CPT with state-of-the-art pre-trained models for understanding and generation.
- Classification: 74.6 average accuracy is achieved by CPTu (B) on the CLUE development set, while CPTug (L) reaches 76.2 and surpasses RoBERTa (L).The best fine-tuning mode differs between base and large models, partly because the generation decoder is shallow in the base model.
- Classification: CPT consistently outperforms BERT, RoBERTa, and ERNIE on base classification results, while large CPT reaches a 74.5 averaged test score.Generative PTMs such as BART can handle discrimination tasks but perform suboptimally compared with CPT.
- Sequence Labeling: CPTu (L) achieves the highest performance on all CWS and NER tasks, while CPTu (B) surpasses base BERT and RoBERTa.The understanding decoder is selected because these tasks rely more on local syntax than on the complex semantics used for generation.
- Machine Reading Comprehension: CPTu obtains comparable or higher MRC results, including 72.3 EM on CMRC development data and 91.1 EM on the DRCD test set.These results exceed or match widely used systems such as RoBERTa, MacBERT, ERNIE, and NEZHA.
- Text Generation: CPTg achieves competitive text-generation performance, with DAE pre-training associated with better results than mT5 and CPM-2 in both base and large versions.The shallow generation decoder may affect long-text generation, but the reported performance gaps remain small.
- Inference efficiency: CPT achieves 1.4×–1.5× speedup over BART in base size and up to 1.7× relative speedup in large size while maintaining comparable generation results in base size.The shallow generation decoder reduces inference cost with minor performance loss.
7 Conclusion
CPT is proposed as a Chinese pre-trained model for both language understanding and generation, with effectiveness demonstrated across Chinese NLU and NLG tasks. Future work will add designs tailored to Chinese properties, including tokenization, pre-training tasks, and model architectures.
- CPT is a Chinese pre-trained model designed for both language understanding and generation.
- Its flexible design allows CPT to be assembled and disassembled in various fashions to exploit the model's potential.
- Experimental results across a wide range of Chinese NLU and NLG tasks show CPT's effectiveness.
- Future work will introduce more Chinese-specific designs, including better tokenization, pre-training tasks, and model architectures.