Source-linked AI summary
Revisiting Pre-Trained Models for Chinese Natural Language Processing
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, Guoping Hu
TL;DR
The paper addresses limited evidence about whether English-developed pre-trained language-model techniques generalize to Chinese. It releases Chinese model series and proposes MacBERT, whose correction-style masking reduces the pre-training–fine-tuning discrepancy; MacBERT improves results on most evaluated tasks, while future masking-ratio selection remains open.
Problem
Most released pre-trained language models are English-based, motivating a systematic examination of whether their techniques generalize to Chinese.
Method
The authors revisit Chinese versions of existing models, release a Chinese model series, and introduce MacBERT with MLM as correction using similar-word masking.
Results
MacBERT achieves significant gains on most Chinese NLP tasks, with experiments spanning eight datasets and detailed ablations of improvement components.
Takeaways & Limitations
The study supports focusing more on the MLM task than NSP-like objectives, whose variants did not show a landslide advantage in the authors’ ablations.
Takeaways & Limitations
The authors leave effective determination of masking ratios beyond heuristic choices for future work.
Abstract
from arXiv · showhide
Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and consecutive variants have been proposed to further improve the performance of the pre-trained language models. In this paper, we target on revisiting Chinese pre-trained language models to examine their effectiveness in a non-English language and release the Chinese pre-trained language model series to the community. We also propose a simple but effective model called MacBERT, which improves upon RoBERTa in several ways, especially the masking strategy that adopts MLM as correction (Mac). We carried out extensive experiments on eight Chinese NLP tasks to revisit the existing pre-trained language models as well as the proposed MacBERT. Experimental results show that MacBERT could achieve state-of-the-art performances on many NLP tasks, and we also ablate details with several findings that may help future research. Resources available: https://github.com/ymcui/MacBERT
1 Introduction
The paper revisits Chinese pre-trained language models to test whether techniques developed largely for English generalize to Chinese, while releasing models and proposing MacBERT. MacBERT uses MLM as correction to reduce pre-training–fine-tuning discrepancy and is evaluated across diverse Chinese NLP tasks.
- Motivation: BERT-based models have become fundamental NLP components, but training large Transformer models is computationally expensive and most released models are English-based.The paper frames Chinese model development as both a language-coverage and accessibility need.
- Contributions: The authors build and release a Chinese pre-trained language model series to examine whether existing approaches generalize beyond English.The study targets Chinese because it is among the world’s most spoken languages.
- Contributions: MacBERT replaces the original MLM task with MLM as correction to mitigate the discrepancy between pre-training and fine-tuning.The model is presented as a new Chinese pre-trained language model.
- Evaluation: Extensive empirical studies evaluate Chinese pre-trained language models across eight popular datasets spanning sentence-level to document-level tasks.The contribution list emphasizes careful analyses of model performance across varied tasks.
- Results: MacBERT achieves significant gains on most tasks against other pre-trained language models, with detailed ablations examining the sources of improvement.The ablations are intended to clarify the composition of the reported improvements.
2 Related Work
The related work reviews BERT and major variants that modify masking, sentence objectives, architecture, or training efficiency. These approaches motivate testing whether their reported techniques transfer effectively to Chinese.
- Scope: The paper positions its review as a test of whether these English-derived conclusions and techniques generalize to other languages.Chinese models are adapted from BERT-based methods for this comparison.
- BERT: BERT pre-trains bidirectional representations with masked language modeling and next sentence prediction.MLM predicts masked tokens from context, while NSP predicts whether one sentence follows another.
- BERT: Whole word masking masks all WordPiece tokens belonging to a word, forcing recovery of the whole word without changing downstream inputs.The method makes the pre-training prediction problem more challenging while adding no downstream-task burden.
- Variants: ERNIE extends masking with entity-level and phrase-level selections, while XLNet addresses the [MASK] discrepancy through permutation language modeling and autoregressive modeling.These methods alter what is masked or how the pre-training objective is formulated.
- Variants: RoBERTa improves BERT through longer training, larger batches, longer sequences, more data, removal of NSP, and dynamic masking.Its design is based on systematic comparisons of BERT components.
- Variants: ALBERT reduces parameters through factorized embeddings and cross-layer sharing, and replaces NSP with sentence-order prediction.Its primary goals are reducing memory requirements and training speed costs.
- Variants: ELECTRA uses a generator–discriminator framework in which the discriminator detects replaced tokens rather than predicting exact masked words.The replacement-detection objective is designed for efficient training.
3 Chinese Pre-trained Language Models
The Chinese model series adapts established pre-training techniques and introduces MacBERT, which changes token masking and replacement while retaining the general BERT pre-training framework.
- Chinese adaptation: Chinese WordPiece tokenization does not split characters into alphabet-like fragments, so the authors use Chinese word segmentation for whole-word masking.Whole-word masking changes token selection during pre-training, not the BERT input tokenizer.
- MacBERT: MacBERT is a simple modification designed to improve fine-tuning by reducing the mismatch between pre-training and fine-tuning.It retains BERT’s pre-training tasks with several modifications.
- MacBERT: MacBERT selects masking candidates with whole-word and N-gram masking, using 40%, 30%, 20%, and 10% for unigram through 4-gram word levels.The candidate-selection strategy combines word-level and span-level masking.
- MacBERT: Instead of [MASK], MacBERT replaces selected tokens with similar words obtained using a word2vec-based synonyms toolkit.If no similar word exists, the method falls back to random replacement.
- MacBERT: MacBERT masks 15% of input words, replacing 80% with similar words, 10% with random words, and retaining 10% unchanged.These replacement proportions define the MLM-as-correction masking procedure.
- MacBERT: For the sentence-level objective, MacBERT uses sentence-order prediction with negative samples formed by swapping consecutive sentences.The paper ablates these modifications to assess their individual contributions.
4 Experimental Setups
The experiments train and compare Chinese pre-trained models using Wikipedia and extended corpora, then evaluate them across eight datasets covering multiple task types and text lengths.
- Pre-training data: The pre-training corpus combines Simplified and Traditional Chinese Wikipedia with extended encyclopedia, news, and question-answering web data.The Wikipedia portion contains about 0.4B words, while the extended data contains 5.4B words.
- Model training: Base-level models inherit vocabulary and weights from official Chinese BERT, whereas large-level models are trained from scratch with the same vocabulary.This distinction applies to the model initialization strategy.
- Model training: RoBERTa and MacBERT use maximum sequence length 512 throughout pre-training because shorter initial training was found insufficient for long-sequence tasks.BERT series models instead use a 128-then-512 training scheme.
- Evaluation: The evaluation covers machine reading comprehension, single-sentence classification, and sentence-pair classification across eight Chinese NLP datasets.The tasks span sentence-level through document-level text lengths.
- Fine-tuning: Fine-tuning keeps dataset hyperparameters fixed while tuning only the initial learning rate from 1e-5 to 5e-5 for each task.The learning rates are tuned on original Chinese BERT, and each experiment is run ten times.
- Model comparison: The comparison includes BERT, whole-word-masked BERT and RoBERTa, ELECTRA, and MacBERT under relatively fair training comparisons.The authors train all listed models except the original Chinese BERT.
- Result reporting: Table 4 reports CMRC 2018 results with averages over 10 independent runs and separately marks best base-level and large-level performances.The table concerns Simplified Chinese machine reading comprehension.
5 Results
The evaluation covers Chinese machine reading comprehension, single-sentence classification, and sentence-pair classification tasks. MacBERT improves substantially on reading comprehension, while gains are moderate on classification tasks and additional general data helps in some settings.
- Machine Reading Comprehension: MacBERT yields significant improvements on all three reading comprehension datasets: CMRC 2018, DRCD, and CJRC.These datasets include span extraction, Traditional Chinese, and Chinese legal-domain question answering settings.
- Machine Reading Comprehension: Additional large-scale Simplified Chinese pretraining improves performance on the Traditional Chinese DRCD dataset.The authors attribute this possibility to shared characters between Simplified and Traditional Chinese and note that character conversion is unnecessary.
- Machine Reading Comprehension: On CJRC, BERT-wwm improves only moderately over BERT, while more general training data improves results when domain data is limited.The paper notes that further domain adaptation is needed for professional Chinese legal text.
- Single Sentence Classification: MacBERT gives moderate improvements on ChnSentiCorp and THUCNews, which already have very high accuracies.These datasets evaluate sentiment classification and long-form news classification across ten domains.
- Sentence Pair Classification: On XNLI, LCQMC, and BQ Corpus, MacBERT outperforms other models with moderate average-score gains, while RoBERTa-wwm-ext-large retains higher peak performance.The authors suggest sentence-pair classification is less sensitive to subtle input differences than reading comprehension.
6 Discussion
The discussion examines which MacBERT pre-training components drive improvements, focusing on MLM masking and comparisons with NSP-like objectives. Results indicate that masking choices matter substantially, while NSP contributes less.
- Ablation Study: Removing any MacBERT component lowers average performance across fine-tuning tasks, indicating that the modifications collectively contribute to improvements.The ablation averages test scores across tasks, averaging EM and F1 before overall averaging.
- Ablation Study: MLM design is more important than NSP, whose variants do not show a decisive advantage over one another.Sentence order prediction performs better than original NSP, but the difference is not salient.
- Investigation on MLM Task: Masking strategies dominated by [MASK] perform worse, showing that pre-training–fine-tuning discrepancy affects downstream performance.The comparison includes partial masking and all-mask variants.
- Investigation on MLM Task: Replacing masked tokens with similar words produces the best performance among the four evaluated masking variants.The alternatives include random replacement, partial masking, and all-mask replacement.
7 Conclusion
The paper revisits Chinese pre-trained language models, introduces MacBERT, and evaluates whether established techniques generalize beyond English. It reports gains across most Chinese NLP tasks and identifies MLM design as more important than NSP-like objectives.
- The study revisits pre-trained language-model techniques in Chinese to assess whether they generalize beyond English.
- MacBERT modifies MLM into a language-correction task to reduce the discrepancy between pre-training and fine-tuning.
- MacBERT provides significant gains on most evaluated Chinese NLP tasks.
- Ablations indicate that MLM deserves more focus than NSP and its variants, which do not show a decisive advantage over one another.
- The authors plan to investigate non-heuristic methods for determining masking ratios.
A.1 XLNet Results on Machine Reading Comprehension Tasks
The appendix evaluates Chinese XLNet models on machine reading comprehension and reports moderate but inconsistent improvements over BERT. The models perform competitively on reading comprehension, while text-classification results remain weaker.
- Setup: XLNet-base and XLNet-mid are trained with a 32,000-token SentencePiece vocabulary on 5.4B training tokens.XLNet-mid and XLNet-base use 24 and 12 layers, respectively, and are trained for 2M and 4M steps.
- Results: XLNet achieves moderate improvements over BERT on CMRC 2018 and DRCD, but gains are inconsistent across subsets.
- Results: The appendix reports XLNet results on CMRC 2018 and DRCD in Tables 10 and 11.
- Results: XLNet-mid reaches near 74% on XNLI, below BERT-base’s 77.8% accuracy.