PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-Training with Whole Word Masking for Chinese BERT

Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang

arXiv:1906.08101v3cs.CLcs.LG

TL;DR

This paper builds and releases Chinese pre-trained language model series, introduces MacBERT with a correction-oriented masking task, and evaluates them across Chinese NLP tasks. MacBERT achieves significant gains on most tasks, while ablations highlight the importance of MLM-related modifications.

  • Problem

    Chinese adaptations of pre-trained language models require systematic baselines, while powerful Transformer-based models are computationally difficult to train and adopt.

  • Method

    The paper creates Chinese BERT, RoBERTa, ELECTRA, and RBT series, and proposes MacBERT, which replaces MLM with MLM as correction using similar-word masking.

  • Results

    Across extensive experiments on Chinese NLP tasks, MacBERT yields significant gains on most tasks and achieves a state-of-the-art F1 of 60% on the CMRC 2018 challenge set.

  • Takeaways & Limitations

    Detailed ablations suggest that MLM-related modifications, especially N-gram masking and similar-word replacement, contribute more than NSP-like tasks to the observed improvements.

Abstract

from arXiv · show

Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and its consecutive variants have been proposed to further improve the performance of the pre-trained language models. In this paper, we aim to first introduce the whole word masking (wwm) strategy for Chinese BERT, along with a series of Chinese pre-trained language models. Then we also propose a simple but effective model called MacBERT, which improves upon RoBERTa in several ways. Especially, we propose a new masking strategy called MLM as correction (Mac). To demonstrate the effectiveness of these models, we create a series of Chinese pre-trained language models as our baselines, including BERT, RoBERTa, ELECTRA, RBT, etc. We carried out extensive experiments on ten Chinese NLP tasks to evaluate the created Chinese pre-trained language models as well as the proposed MacBERT. Experimental results show that MacBERT could achieve state-of-the-art performances on many NLP tasks, and we also ablate details with several findings that may help future research. We open-source our pre-trained language models for further facilitating our research community. Resources are available: https://github.com/ymcui/Chinese-BERT-wwm

I. INTRODUCTION

The paper addresses the cost and language imbalance of pre-trained language modeling by building Chinese model series and introducing MacBERT. It evaluates these models across Chinese NLP tasks and releases them for research use.

  • BERT and its variants have become fundamental components of NLP through large-scale pre-training and contextual representations.
  • Training large Transformer-based language models requires expensive, high-memory computing devices.
  • Most released pre-trained language models are English-based, leaving relatively few powerful models for other languages.
  • The authors create and publicly release Chinese pre-trained model series, including baselines for future studies.
  • MacBERT mitigates the pre-training–fine-tuning gap by masking words with similar words and is evaluated on ten Chinese NLP datasets.
  • The paper also develops smaller RBT models to examine performance–efficiency trade-offs for real-life applications.

B. ERNIE

This section reviews representative pre-trained language models and their strategies for improving efficiency, masking, parameterization, and language-model training.

  • B. ERNIE: ERNIE improves BERT masking through entity-level and phrase-level masking strategies.
  • B. ERNIE: XLNet addresses the masking discrepancy with permutation language modeling and a two-stream self-attention mechanism.
  • B. ERNIE: RoBERTa strengthens BERT through longer training, larger batches and sequences, more data, and removal of next sentence prediction.
  • B. ERNIE: ALBERT reduces parameters using factorized embedding parameterization and cross-layer parameter sharing, while replacing NSP with SOP.

F. ELECTRA

The Chinese model series adapts BERT, RoBERTa, and ELECTRA while retaining compatible inputs, and uses whole-word masking for Chinese segmentation.

  • F. ELECTRA: ELECTRA uses a generator–discriminator framework in which the discriminator detects tokens replaced by the generator.
  • F. ELECTRA: The Chinese models adapt BERT, RoBERTa, and ELECTRA without changing the nature of their inputs or fine-tuning procedures.
  • F. ELECTRA: Table I compares the models by encoding type, masking strategy, sentence objective, and pre-training framework.
  • F. ELECTRA: Chinese whole-word masking uses traditional Chinese word segmentation because WordPiece tokenization does not split Chinese words into alphabet-like fragments.
  • F. ELECTRA: RoBERTa retains paired inputs for pre-training despite not using the NSP task, supporting sentence-pair classification and reading comprehension.

C. RBT Series

The RBT series provides smaller Chinese RoBERTa-based models, while MacBERT replaces artificial masking with a correction task and combines multiple masking choices.

  • C. RBT Series: RBT3, RBT4, RBT6, and RBTL3 reduce Transformer depth to make pre-trained models more accessible for real-life applications.
  • C. RBT Series: RBTL3 and RBT6 have comparable parameter sizes, enabling comparison between wider-and-shorter and thinner-and-taller architectures.
  • MacBERT: MacBERT combines MLM as correction with sentence order prediction in its pre-training architecture.
  • MacBERT: MLM as correction reframes masking as text correction, replacing corrupted words with correct words without predefined masking tokens.
  • MacBERT: MacBERT selects masking candidates with whole-word and N-gram masking, then mainly replaces them with similar words.

B. Sentence Order Prediction

MacBERT combines MLM as correction with sentence order prediction (SOP) in a Transformer architecture. SOP replaces BERT’s less-effective NSP objective by distinguishing correctly ordered consecutive texts from texts whose order is switched.

  • B. Sentence Order Prediction: SOP replaces NSP by classifying consecutive texts as positive examples and texts with switched order as negative examples.The paper adopts SOP because NSP is considered too easy and less effective.
  • B. Sentence Order Prediction: MacBERT constructs an input sequence by concatenating two sequences with [CLS] and [SEP] tokens.The resulting sequence is converted into contextualized representations through embeddings and a consecutive Transformer stack.
  • B. Sentence Order Prediction: The Mac task predicts only the replaced positions, selecting k = ⌊N × 15%⌋ tokens from the contextual representation.The selected representations form Hm before vocabulary-space prediction.
  • B. Sentence Order Prediction: MacBERT projects replaced-token representations into the vocabulary space and optimizes the prediction with standard cross-entropy loss.The projection uses the word embedding matrix because embedding and hidden dimensions are identical.
  • B. Sentence Order Prediction: For SOP, MacBERT uses the [CLS] representation for label prediction, and the overall loss combines the Mac and SOP objectives.The SOP classifier is trained with cross-entropy loss.

V. EXPERIMENTAL SETUPS

The experimental setup uses Chinese Wikipedia together with extended Chinese data, different model scales, and documented training configurations. The authors provide training details for the model series and distinguish models trained with extended data.

  • A. Data Processing: Chinese Wikipedia data comprise about 0.4B words after cleaning, while extended training data add 5.4B words from encyclopedia, news, and question-answering websites.Both Simplified and Traditional Chinese are retained in the Wikipedia dump.
  • B. Setups for Pre-Trained Language Models: Table III reports the training details of the Chinese pre-trained language models, with extended-data models omitted because their other parameters match non-extended counterparts.The omitted models are identified by the ‘ext’ designation.
  • B. Setups for Pre-Trained Language Models: The base-level models use 12 Transformer layers with hidden dimension 768, while large-level models use 24 layers with hidden dimension 1024.The base-level model inherits the official Chinese BERT-base vocabulary and weights; the large-level model is trained from scratch with the same vocabulary.
  • B. Setups for Pre-Trained Language Models: BERT models are trained first with maximum sequence length 128 and then 512, whereas other models use maximum length 512 throughout pre-training.The authors changed this scheme because they found it insufficiently adapted to long-sequence tasks.

C. Setups for Fine-tuning Tasks

The study evaluates Chinese pre-trained language models across ten datasets spanning machine reading comprehension, single-sentence classification, and sentence-pair classification. Fine-tuning uses controlled hyperparameters, repeated runs, and a selected comparison set for fairness.

  • C. Setups for Fine-tuning Tasks: The evaluation covers ten Chinese datasets across machine reading comprehension, single-sentence classification, and sentence-pair classification.The tasks range from sentence-level to document-level text lengths.
  • C. Setups for Fine-tuning Tasks: For each dataset, the authors keep fine-tuning hyperparameters fixed and tune only the initial learning rate from 1e-5 to 5e-5.The best initial learning rate is selected using the original Chinese BERT.
  • C. Setups for Fine-tuning Tasks: Each experiment is run ten times to improve result reliability.The authors report average scores for the CMRC 2018 and DRCD results in Table V.
  • C. Setups for Fine-tuning Tasks: The same initial learning-rate settings are used for most models, while ELECTRA uses 1e-4 for base-level and 5e-5 for large-level models.The ELECTRA settings follow the recommendation associated with that model.
  • C. Setups for Fine-tuning Tasks: Comparisons include BERT, BERT-wwm, BERT-wwm-ext, RoBERTa-wwm-ext, RoBERTa-wwm-ext-large, ELECTRA, and MacBERT.The authors train all listed models except the original Chinese BERT.

A. Machine Reading Comprehension

The models are evaluated on Chinese machine reading comprehension datasets spanning simplified, traditional, and legal-domain text. MacBERT improves performance across all reading-comprehension datasets, while additional pretraining data and domain adaptation affect results.

  • Datasets: CMRC 2018 and DRCD are span-extraction datasets, while CJRC includes yes/no, no-answer, and span-extraction questions.DRCD uses Traditional Chinese, and CJRC contains Chinese legal judgment documents.
  • Results: MacBERT yields significant improvements on all three reading comprehension datasets.The paper reports this result in comparisons summarized in Tables V and VI.
  • Results: 60% F1 on the CMRC 2018 challenge set is achieved by MacBERT-large.The challenge set requires deeper text understanding.
  • Results: Additional pretraining data improves results, including on Traditional Chinese DRCD without converting its characters to simplified Chinese.The paper attributes this possibility to many identical characters being shared between simplified and traditional Chinese.
  • Results: CJRC shows moderate BERT-wwm gains, while increased general pretraining data improves performance when domain-specific data is limited.The paper identifies further domain adaptation as needed for professional legal text.

C. Sentence Pair Classification

The paper compares MacBERT and related Chinese pretrained models across sentence-pair classification and small-model settings. MacBERT is strongest on XNLI and OCNLI, while task sensitivity and model size shape the observed gains.

  • Sentence Pair Classification: XNLI, LCQMC, BQ Corpus, and OCNLI require two input sequences and prediction of their relation.These are the sentence-pair classification benchmarks used in the study.
  • Sentence Pair Classification: MacBERT produces relatively consistent and significant improvements on XNLI and OCNLI.Its improvements on LCQMC and BQ Corpus are moderate, with RoBERTa-wwm-ext-large retaining the best peak performance there.
  • Task Differences: MacBERT gains are larger on machine reading comprehension than on classification tasks.The paper suggests reading comprehension benefits from identifying exact answer spans and learning word boundaries.
  • Model Scale: MacBERT-base generally improves more than MacBERT-large.The paper links this pattern to BERT-base initialization and the higher baseline scores of large models.
  • Small Models: RBT models lose performance because they use fewer parameters, but classification degrades less than reading comprehension.Among similarly sized models, RBT6 substantially outperforms RBTL3, favoring a thin-and-tall design.
  • Ablation: Removing any MacBERT component lowers average performance, with N-gram masking and similar-word replacement being most effective.N-gram masking helps classification more, whereas similar-word replacement benefits reading comprehension more.

B. Investigation on MLM Task

The study investigates how masked-token replacement affects Chinese pretrained-model performance. Mask-based variants perform worse, random replacement improves results, and MacBERT’s similar-word replacement performs best among the tested variants.

  • Experimental Setup: The investigation varies replacement strategies while retaining the original 15% masking rate, including identity, random-word, partial-mask, and all-mask variants.The remaining 90% of selected tokens are divided among the replacement conditions.
  • Findings: Partial-mask and all-mask strategies produce worse performance than alternatives that rely less on [MASK].The paper interprets this as evidence of a pretraining–fine-tuning discrepancy.
  • Findings: Randomly replacing all 90% of masked tokens improves results consistently over [MASK]-dependent strategies.The paper also suggests random words may improve denoising ability, although they can reduce sentence naturalness.
  • MacBERT: MacBERT replaces masked tokens with similar words, turning the objective into a more natural grammar-correction task.Similar-word replacement preserves fluency better than random replacement and reduces the pretraining–fine-tuning discrepancy.
  • Results: MacBERT achieves the best performance among the four replacement variants on CMRC 2018 and DRCD.Figure 2 reports the comparison across pretraining steps for both datasets.

C. Analyses on Chinese Spell Check

MacBERT consistently outperforms the compared Chinese pre-trained models on spell-checking across training-data percentages. The authors attribute its additional gain to reducing the mismatch between pre-training and fine-tuning, while noting a limitation of synonym-only replacement.

  • Experimental setup: The experiment evaluates detection-level and correction-level scores on SIGHAN-15 using varying amounts of training data.SIGHAN-15 contains a 3.1K-instance training set and a 1.1K-instance test set.
  • Experimental results: MacBERT consistently improves over BERT-wwm-ext, RoBERTa-wwm-ext, and ELECTRA-base across different percentages of SIGHAN-15 training data.The models share pre-training data, and each was fine-tuned five times with averaged sentence-level F1 results.
  • Model comparison: ELECTRA shows a relatively larger gap from the other models on correction-level results than on detection-level results.The authors relate this pattern to ELECTRA’s replaced-token detection objective, which predicts whether tokens were altered rather than restoring the original token.
  • Model comparison: Comparing MLM and Mac, the authors report that MacBERT reduces the pre-training–fine-tuning discrepancy and gains further performance.Mac replaces masked tokens with similar words, making the pre-training task more like grammar or spelling correction.
  • Limitation: MacBERT’s improvement is smaller with less training data because synonym replacement covers only a small proportion of real spell-check errors.The authors identify this as a limitation of the Mac task’s replacement strategy.
Loading 1906.08101v3…