PaperlayerView exact revision on arXiv

Source-linked AI summary

ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations

Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, Yonggang Wang

arXiv:1911.00720v1cs.CL

TL;DR

Small-unit Chinese text encoding can omit semantic information carried by larger character combinations, while segmentation-based adaptations may introduce error propagation. ZEN adds a multi-layer n-gram encoder to BERT and integrates n-gram representations with character encoding, achieving state-of-the-art results on most evaluated tasks with less resource than other published encoders.

  • Problem

    Small-unit tokenization can lose larger-granularity information important to Chinese semantics, where valid n-gram boundaries matter and explicit word boundaries are absent.

  • Method

    ZEN extends BERT with a multi-layer encoder that extracts, represents, and integrates n-grams into character-level encoding without changing downstream output format.

  • Results

    ZEN outperformed BERT in all seven evaluated tasks under both random-initialization and pretrained settings, and achieved state-of-the-art results on most tasks with less resource than other encoders.

  • Takeaways & Limitations

    ZEN explicitly learns larger-granularity text information while remaining complementary to prior approaches and effective with limited training data.

  • Takeaways & Limitations

    ZEN was not as strong as ERNIE 2.0 and NEZHA on the NLI task, whose larger models and corpora provide more prior knowledge for inference.

Abstract

from arXiv · show

The pre-training of text encoders normally processes text as a sequence of tokens corresponding to small text units, such as word pieces in English and characters in Chinese. It omits information carried by larger text granularity, and thus the encoders cannot easily adapt to certain combinations of characters. This leads to a loss of important semantic information, which is especially problematic for Chinese because the language does not have explicit word boundaries. In this paper, we propose ZEN, a BERT-based Chinese (Z) text encoder Enhanced by N-gram representations, where different combinations of characters are considered during training. As a result, potential word or phase boundaries are explicitly pre-trained and fine-tuned with the character encoder (BERT). Therefore ZEN incorporates the comprehensive information of both the character sequence and words or phrases it contains. Experimental results illustrated the effectiveness of ZEN on a series of Chinese NLP tasks. We show that ZEN, using less resource than other published encoders, can achieve state-of-the-art performance on most tasks. Moreover, it is shown that reasonable performance can be obtained when ZEN is trained on a small corpus, which is important for applying pre-training techniques to scenarios with limited data. The code and pre-trained models of ZEN are available at https://github.com/sinovation/zen.

1 Introduction

Standard Chinese text encoders process characters as small units, potentially losing semantic information carried by valid n-grams. ZEN addresses this gap while avoiding dependence on potentially erroneous segmentation or entity-recognition results.

  • 1 Introduction: Chinese semantics are greatly affected by recognizing valid n-grams, but standard small-unit encoding can lose information carried by larger text units.Here, “valid” n-grams are proper chunks or phrases frequently used in running text.
  • 1 Introduction: Existing Chinese BERT adaptations use weakly supervised word information or external knowledge, limiting how they preserve the original BERT structure.Whole-word masking, for example, uses an existing segmenter to identify possible words before masking.
  • 1 Introduction: Incorrect word segmentation or entity recognition can propagate errors into pre-training and negatively affect encoder generalization.
  • 1 Introduction: ZEN instead incorporates larger-granularity n-gram information into character-based encoders while retaining character-level outputs for downstream tasks.

2 ZEN

ZEN extends BERT with a multi-layer n-gram encoder that extracts candidate n-grams and integrates their representations into character processing. The resulting model preserves BERT-compatible character-level outputs while explicitly modeling larger text units.

  • 2 ZEN: The architecture retains BERT’s character encoder and training objectives while adding explicit n-gram representations.Figure 1 identifies the BERT character encoder, the n-gram encoder, masked language modeling, next sentence prediction, and the n-gram preparation process.
  • 2.1 N-gram Extraction: ZEN extracts n-grams from a lexicon, records their positions with a matching matrix, and processes them alongside each input character.The lexicon may come from predefined dictionaries or n-gram lists extracted by unsupervised methods.
  • 2.2 Encoding N-grams: A multi-layer Transformer n-gram encoder models interactions among n-grams at layers aligned with the BERT character encoder.Unlike sequence encoders, it omits n-gram positions because the n-grams are treated equally without sequential order.
  • 2.3 Representing N-grams in Pre-training: ZEN combines each character embedding with embeddings of its associated n-grams through elementwise addition before sending the enhanced representation to the next layer.The combination is performed layer by layer through the matching matrix, except at the final BERT layer.

3 Experiment Settings

The experiments pre-train ZEN and comparison encoders on Chinese Wikipedia, then evaluate them on seven Chinese NLP tasks using standardized datasets and task settings.

  • 3.1 Tasks and Datasets: Chinese Wikipedia supplies the 474M-token pre-training corpus, while traditional characters are converted to simplified ones and English letters are lowercased.
  • 3.1 Tasks and Datasets: Seven tasks cover word segmentation, POS tagging, NER, document classification, sentiment analysis, sentence pair matching, and natural language inference.The datasets include MSR, CTB5, MSRA, THUCNews, ChnSentiCorp, LCQMC, and Chinese XNLI.
  • 3.2 Implementation: N-gram lexicons are extracted from the same corpus, filtered by frequency thresholds of 5–40, and range from 179K to 64K entries.The main experiments use a cutoff of 15, yielding 104K n-grams.
  • 3.2 Implementation: ZEN retains BERT’s 12-layer, 12-head backbone and masking objectives while using a six-layer n-gram encoder with up to 128 n-grams per instance.
  • 3.2 Implementation: Table 2 compares random-initialized and pretrained models across seven tasks, distinguishing BERT base from large-model reference systems.

4 Experimental Results

Across seven tasks, ZEN consistently improves over BERT and achieves strong comparisons with other pretrained encoders, while its advantage varies by task and resource scale.

  • 4.1 Overall Performance: ZEN outperforms BERT in all seven tasks under both random-initialized and pretrained settings.
  • 4.1 Overall Performance: Token-level tasks show larger ZEN improvements than sentence-level tasks because n-gram boundaries guide character labeling and frequent n-grams often carry key semantic information.
  • 4.1 Overall Performance: ZEN achieves state-of-the-art performance on many tasks without external information, despite competing with encoders using larger models, more data, or additional pre-training signals.
  • 4.1 Overall Performance: ZEN outperforms BERT-wwm on every task with reported BERT-wwm results, indicating an advantage for explicit n-gram integration over whole-word masking.
  • 4.1 Overall Performance: ZEN exceeds ERNIE 2.0 (B) on SA (TEST) and SPM (TEST), while remaining competitive on SA, SPM, and NLI despite substantially fewer resources.
  • 4.1 Overall Performance: ZEN is weaker than ERNIE 2.0 and NEZHA on NLI, where larger models and larger corpora provide more prior knowledge for inference.

5 Analyses

ZEN consistently outperforms BERT across probing analyses, while its performance depends on balanced n-gram lexicon thresholds and modest per-instance n-gram counts. Layerwise visualizations further show context-sensitive selection and increasing emphasis on longer valid n-grams in higher layers.

  • 5.1 Effects of Pretraining Epochs: ZEN outperforms BERT on CWS and SA at comparable pretraining stages under both random and pretrained initialization.The comparison uses pretrained models fine-tuned at selected epochs.
  • 5.2 Effects of N-gram Extraction Threshold: ZEN performs best on both CWS and SA when the n-gram extraction threshold is 15, with thresholds below or above 15 degrading performance.The threshold analysis ranges from 0 to 40 and uses at most 128 n-grams during pretraining.
  • 5.2 Effects of N-gram Extraction Threshold: A maximum of 32 n-grams per input gives a good performance–computation tradeoff, although using more n-grams provides a small additional gain.The tested maximum ranges from 0 to 128, with the x-axis shown on a base-2 logarithmic scale.
  • 5.3 Visualization of N-gram Representations: ZEN assigns higher weights to valid, contextually appropriate n-grams and increasingly emphasizes longer valid n-grams in higher layers.Examples include distinguishing 提高 from 会提高 and 速度 from the contextually inappropriate 高速, while longer phrases receive stronger higher-layer weights.

6 Related Work

Prior Chinese encoders incorporate word-level information through masking, continual pretraining, optimization, or external knowledge. These approaches collectively support processing Chinese text at larger granularities.

  • 6 Related Work: Chinese pretraining studies enhance BERT with word-level information because words carry important linguistic information in Chinese.The cited approaches include multi-level masking, continual multitask pretraining, whole-word masking, specialized optimization, and prior knowledge.
  • 6 Related Work: ERNIE uses multi-level masking and later continual multitask pretraining, while other work adapts BERT through whole-word masking, optimization, or prior knowledge.ERNIE 2.0 adds more parameters through continual pretraining with multitask learning.

7 Conclusion

ZEN extends BERT with an n-gram encoder that integrates larger-granularity character combinations into Chinese representation learning. Experiments report state-of-the-art results with fewer resources, effectiveness with limited data, and complementarity with prior approaches.

  • 7 Conclusion: ZEN extends a BERT backbone with a Transformer-style encoder that extracts, encodes, and integrates n-grams for each Chinese input.This provides an alternative way to learn larger-granularity text while retaining the character encoder.
  • 7 Conclusion: ZEN achieves state-of-the-art results on several NLP tasks while using the BERT base model, less training data, and no external knowledge compared with existing Chinese encoders.Further analyses characterize ZEN as efficient and capable of learning with limited data.
  • 7 Conclusion: ZEN’s method for incorporating word information is complementary to previous approaches and may be combined with them or applied to other languages.The paper presents this as a potential benefit rather than an experimentally established result.
Loading 1911.00720v1…