Source-linked AI summary
ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Guoyin Wang, Xiang Ao, Qing He, Fei Wu, Jiwei Li
TL;DR
Chinese pretraining models omit glyph and pinyin information that encode Chinese character semantics, pronunciation, and syntactic distinctions. ChineseBERT integrates glyph, pinyin, and character embeddings into large-scale pretraining, achieving stronger results across Chinese NLP tasks and performing better with less training data. Its corpus may be less suitable for sentence pair matching than the BERT-wwm pretraining data.
Problem
Chinese pretraining models omit glyph-based and pinyin-based information, despite their roles in representing character semantics, pronunciation, and heteronym distinctions.
Method
ChineseBERT combines character embeddings with font-derived glyph embeddings and CNN-derived pinyin embeddings in a fusion embedding for large-scale pretraining.
Results
ChineseBERT achieves significant performance gains across a wide range of Chinese NLP tasks and performs better than vanilla pretrained models with less training data.
Takeaways & Limitations
Glyph and pinyin embeddings serve as strong regularizers for semantic modeling in Chinese and support improved performance with less training data.
Takeaways & Limitations
The corpus is more suitable for BERT-wwm pretraining than for ChineseBERT in sentence pair matching.
Abstract
from arXiv · showhide
Recent pretraining models in Chinese neglect two important aspects specific to the Chinese language: glyph and pinyin, which carry significant syntax and semantic information for language understanding. In this work, we propose ChineseBERT, which incorporates both the {\it glyph} and {\it pinyin} information of Chinese characters into language model pretraining. The glyph embedding is obtained based on different fonts of a Chinese character, being able to capture character semantics from the visual features, and the pinyin embedding characterizes the pronunciation of Chinese characters, which handles the highly prevalent heteronym phenomenon in Chinese (the same character has different pronunciations with different meanings). Pretrained on large-scale unlabeled Chinese corpus, the proposed ChineseBERT model yields significant performance boost over baseline models with fewer training steps. The porpsoed model achieves new SOTA performances on a wide range of Chinese NLP tasks, including machine reading comprehension, natural language inference, text classification, sentence pair matching, and competitive performances in named entity recognition. Code and pretrained models are publicly available at https://github.com/ShannonAI/ChineseBert.
1 Introduction
ChineseBERT addresses missing glyph and pinyin information in Chinese pretraining by integrating both into a character representation. It reports stronger performance than baselines across many Chinese NLP tasks, including new state-of-the-art results in several tasks.
- Current large-scale Chinese pretraining omits glyph-based and pinyin-based information specific to Chinese.Chinese character glyphs encode semantic information, while pinyin represents pronunciation and can support semantic and syntactic modeling.
- Glyphs can encode shared semantics through visual character forms, such as the water radical in characters related to water.
- Pinyin distinguishes heteronyms whose identical character forms have different pronunciations and meanings.The same character can have pronunciation-dependent semantic and part-of-speech differences that glyph and character-ID embeddings cannot distinguish.
- ChineseBERT combines character, glyph, and pinyin embeddings into a fusion embedding during large-scale pretraining.Glyph embeddings use different fonts, while pinyin embeddings model meanings associated with the same character form.
- ChineseBERT achieves significant performance gains over baselines with less training data and fewer epochs across a broad range of Chinese NLP tasks.It reaches new state-of-the-art results in machine reading comprehension, natural language inference, text classification, and sentence pair matching, with competitive named entity recognition and word segmentation results.
2 Related Work
Related work adapts pretrained modeling to Chinese linguistic characteristics through character units, multi-granularity masking, and glyph representations. ChineseBERT builds on this line by incorporating glyph information into large-scale pretraining alongside its broader Chinese-specific design.
- Chinese pretraining research adapts English-oriented methods to Chinese syntax, lexicon, and pronunciation.
- Prior Chinese models use character-level units and masking strategies at character, phrase, and entity granularity.
- Glyph information has been modeled through indexed radicals and character images to capture Chinese character semantics.
- Earlier visual-glyph approaches showed inconsistent gains across tasks, while later work evaluated glyph embeddings more broadly.
3 Model
ChineseBERT derives glyph and pinyin embeddings for each Chinese character and fuses them with the character embedding before BERT processing. The model uses visual fonts, CNN-based pinyin encoding, and a fully connected fusion layer.
- Model overview: ChineseBERT represents each character with char, glyph, and pinyin embeddings that are combined into a fusion embedding.The fusion embedding is added to positional embeddings before entering the BERT model.
- Input: The fusion embedding is added to learnable absolute positional embeddings and supplied to the BERT layers.The model uses character-granularity embeddings and applies both Whole Word Masking and Char Masking during pretraining.
- Glyph Embedding: Glyph embeddings are obtained from three 24 × 24 font images that are concatenated, flattened, and passed through a fully connected layer.The fonts are FangSong, XingKai, and LiShu, producing a 2,352-dimensional flattened input.
- Pinyin Embedding: Pinyin embeddings encode pronunciation sequences with a width-2 CNN followed by max-pooling.Special tokens represent tones, and max-pooling makes the output dimensionality independent of sequence length.
- Fusion Embedding: The fusion layer concatenates the three D-dimensional embeddings and maps the resulting 3D-dimensional vector to D dimensions.
- Output: The model outputs a contextualized representation for each input Chinese character.
4 Pretraining Setup
ChineseBERT is pretrained on filtered Common Crawl data using character- and word-level masking, alternating packed and single-sentence inputs. The setup includes dynamic masking and base/large model configurations.
- Data: 4B Chinese characters from filtered Common Crawl data are used for pretraining after retaining about 10% high-quality data.Preprocessing removes data with too much English text and filters HTML tags.
- Masking: Char Masking randomly masks individual Chinese characters, while Whole Word Masking targets multi-character Chinese words identified using LTP word boundaries.Whole Word Masking addresses cases where character-level prediction may be too easy for multi-character words.
- Masking: ChineseBERT uses Whole Word Masking 90% of the time and Char Masking 10% of the time, with a 15% masking probability.Selected items are masked, randomly replaced, or kept according to the stated 80%/10%/10% distribution, with dynamic masking preventing duplicate instances.
- Input construction: Pretraining alternates between packed inputs and single inputs, using packed inputs with probability 0.9 and single inputs with probability 0.1.Packed inputs concatenate multiple sentences up to a maximum length of 512 to model long- and short-term dependencies.
- Model configurations: The base and large setups contain 12 and 24 Transformer layers, with input dimensionalities of 768 and 1,024, respectively.They use 12 and 16 attention heads per layer, respectively.
5 Experiments
ChineseBERT is evaluated across six Chinese NLP task families against established pretrained baselines, with reported gains varying by task and dataset. Results include strong performance in reading comprehension, NLI, classification, matching, NER, and word segmentation, while some comparisons expose dataset-specific differences.
- Experimental setup: ChineseBERT is evaluated on machine reading comprehension, natural language inference, text classification, sentence pair matching, named entity recognition, and Chinese word segmentation.The experiments compare the model with ERNIE, BERT-wwm, and MacBERT, using task-specific datasets.
- Experimental setup: ChineseBERT uses fewer training steps than baseline models despite being trained from scratch rather than initialized from pretrained BERT.The authors attribute the initialization difference to the additional glyph and pinyin components, which change the model structure.
- Machine Reading Comprehension: ChineseBERT yields significant performance boosts on both CMRC and CJRC machine reading comprehension datasets.On CJRC, the improvement in EM is larger than the improvement in F1, indicating better exact answer-span detection.
- Natural Language Inference: ChineseBERT achieves the best performances for both base and large setups on XNLI.XNLI evaluates whether a hypothesis is entailed by, neutral to, or contradicts a premise, using accuracy for comparison.
- Text Classification: ChineseBERT’s classification gains are marginal on ChnSentiCorp and THUCNews but exceed all other models on TNEWS.The first two datasets already yield accuracy above 95% with vanilla BERT, whereas TNEWS is more difficult.
- Sentence Pair Matching and Word Segmentation: ChineseBERT generally outperforms MacBERT on LCQMC, slightly underperforms BERT-wwm there, and outperforms BERT-wwm and RoBERTa-wwm on both word-segmentation datasets.The authors hypothesize that BQ Corpus is better matched to BERT-wwm’s pretraining data than to ChineseBERT’s.
- Named Entity Recognition: ChineseBERT significantly outperforms BERT and RoBERTa on NER F1, with especially high recall gains despite a slight precision loss for the base model.The reported NER datasets are OntoNotes 4.0 and Weibo.
6 Ablation Studies
The ablations examine how glyph and pinyin embeddings affect ChineseBERT and how performance changes with training-data size. Removing either embedding hurts performance, removing both has the largest impact, and ChineseBERT performs better across all tested data sizes.
- The ablation compares models with matched datasets, training steps, and model sizes, varying whether glyph and pinyin embeddings are included.The -glyph, -pinyin, and -glyph-pinyin setups isolate the contributions of the two embedding types against character-ID representations.
- Removing either glyph or pinyin embeddings degrades performance, while removing both causes the greatest decline in F1.The combined removal produces an F1 drop of about 2 points.
- ChineseBERT performs better across all tested training-data sizes, with larger gains above 30% of the training data.Improvements are slight below 30%, while insufficient data limits training of the glyph and pinyin embeddings.
7 Conclusion
ChineseBERT incorporates glyph and pinyin information into large-scale Chinese pretraining to capture character-form semantics and disambiguate polyphonic characters. It improves performance across a wide range of Chinese NLP tasks and outperforms vanilla pretrained models with less training data.
- ChineseBERT leverages glyph and pinyin information to capture semantics from character forms and disambiguate polyphonic Chinese characters.
- ChineseBERT achieves significant performance improvements across a wide range of Chinese NLP tasks.
- ChineseBERT performs better than vanilla pretrained models with less training data.The paper attributes this pattern to glyph and pinyin embeddings serving as strong regularization for semantic modeling.
- Future work involves training a large-size version of ChineseBERT.