Source-linked AI summary
NEZHA: Neural Contextualized Representation for Chinese Language Understanding
Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, Qun Liu
TL;DR
Pre-trained language models have shown success on NLU tasks, while most existing models are trained on English corpora and Chinese models use differing approaches. NEZHA adapts BERT-based pre-training to Chinese with functional relative positional encoding and other training techniques, and the report says it achieves state-of-the-art performance on several Chinese NLU tasks.
Problem
Most pre-trained language models are learned from English corpora, while Chinese models differ in their masking strategies and pre-training tasks.
Source: §1 · p. 1Method
NEZHA is based on BERT and trained on Chinese text, using functional relative positional encoding, Whole Word Masking, mixed precision training, and the LAMB optimizer.
Source: §1 · p. 2Results
NEZHA achieves state-of-the-art performance on several Chinese natural language understanding tasks.
Source: §4 · p. 7Takeaways & Limitations
The authors report that functional relative positional encoding yields notable improvement over other positional encodings.
Source: §4 · p. 7Some comparison results were copied from prior papers, and differences in experimental settings or fine-tuning methods may make the comparison not entirely fair.
Source: §3.2 · p. 6
Abstract
from arXiv · showhide
The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora. In this technical report, we present our practice of pre-training language models named NEZHA (NEural contextualiZed representation for CHinese lAnguage understanding) on Chinese corpora and finetuning for the Chinese NLU tasks. The current version of NEZHA is based on BERT with a collection of proven improvements, which include Functional Relative Positional Encoding as an effective positional encoding scheme, Whole Word Masking strategy, Mixed Precision Training and the LAMB Optimizer in training the models. The experimental results show that NEZHA achieves the state-of-the-art performances when finetuned on several representative Chinese tasks, including named entity recognition (People's Daily NER), sentence matching (LCQMC), Chinese sentiment classification (ChnSenti) and natural language inference (XNLI).
1 Introduction
NEZHA adapts BERT-based pre-training to Chinese text and combines a functional relative positional encoding scheme with established training techniques. The report also evaluates Chinese pre-trained models and training factors across several NLU tasks.
1 Introduction
- Pre-trained language models learn contextualized representations from large text collections and can be fine-tuned for downstream NLP tasks.The cited applications include sentiment analysis, machine reading comprehension, sentence matching, named entity recognition, and natural language inference.Source: §1 · p. 1
- Chinese pre-trained models include Google’s BERT, ERNIE-Baidu, and BERT-WWM; their MLM masking strategies range from individual tokens to whole words or phrases.ERNIE-Baidu 2.0 also incorporates Token-Document Relation Prediction and Sentence Reordering.Source: §1 · p. 1
- NEZHA uses functional relative positional encoding in self-attention, alongside Whole Word Masking, mixed precision training, and the LAMB optimizer.The relative positions are encoded by predefined functions without trainable parameters.Source: §1 · p. 2
- The report systematically studies large-scale Chinese pre-training, evaluates models on Chinese NLU tasks, and assesses positional encoding, masking, corpus sources, and sequence length.The authors also say they will release NEZHA models and source code.Source: §1 · p. 2
2 Pre-training NEZHA Models
The section outlines BERT and positional encoding before presenting NEZHA’s functional relative encoding and its pre-training techniques. It covers whole-word masking, mixed-precision training, and LAMB optimization.
2 Pre-training NEZHA Models
- The section introduces BERT preliminaries, positional encoding, NEZHA’s relative encoding, and the three additional pre-training techniques.These are the topics organized across the section’s subsections.Source: §2 · p. 2
2.1 Preliminaries: BERT Model & Positional Encoding
- BERT is a stack of Transformer encoders trained with masked language modeling and next-sentence prediction; self-attention uses learned projections and softmax scores.The section contrasts absolute position embeddings with relative position information incorporated into attention computations.Source: §2.1 · pp. 2–3
2.2 Functional Relative Positional Encoding
- NEZHA uses fixed sinusoidal functions to encode relative positions in attention outputs and scores, with different wavelengths across dimensions.The wavelengths progress geometrically from 2π to 10000 · 2π; the authors choose fixed functions partly because they may support extrapolation to longer sequences.Source: §2.2 · p. 3
2.3 Whole Word Masking
- Whole Word Masking masks all characters belonging to a Chinese word together, with Jieba used to segment words for NEZHA’s training data.Roughly 12% of characters are masked and 1.5% are randomly replaced.Source: §2.3 · p. 3
2.4 Mixed Precision Training
- Mixed-precision training uses FP16 for forward and backward passes while retaining FP32 master weights and updating them with FP32 gradients.The report says this technique can speed training by 2–3 times and reduce memory use, allowing a larger batch size.Source: §2.4 · pp. 3–4
2.5 LAMB Optimizer
- The LAMB optimizer is designed for large-batch synchronous distributed training and reduced BERT training time from 3 days to 76 minutes.It uses a general adaptation strategy for large batches; the cited report describes batch sizes above 30k.Source: §2.5 · p. 4
3 Experiments
The experiments compare NEZHA with Chinese pretrained models on downstream NLU tasks and test how training data and techniques affect NEZHA. Results favor NEZHA or ERNIE-Baidu 2.0 among BASE and LARGE models, while ablations highlight relative positional encoding.
3 Experiments
- The reported pretraining techniques are not limited to Chinese and can be readily applied to other languages.The experiments report pretraining on Chinese text and finetuning on Chinese downstream tasks.Source: §3 · p. 4
3.1 Experimental Setting
- Pretraining uses Chinese Wikipedia, Baidu Baike, and Chinese News, with both character-level masking and whole word masking data versions.For whole word masking, Jieba identifies word boundaries; masked Chinese characters comprise roughly 12% of each sample, with 1.5% randomly replaced.Source: §3.1 · p. 4
- NEZHA is trained from scratch with mixed-precision FP16 on distributed Huawei Cloud V100 GPUs, using separate batch sizes and learning-rate schedules for BASE and LARGE models.The total batch sizes are 14,400 for BASE and 5,120 for LARGE.Source: §3.1 · p. 4
3.2 Experimental Results
- The evaluation compares Chinese pretrained models on reading comprehension, natural language inference, sentence matching, named entity recognition, and sentiment classification.The compared models include Chinese Google BERT, BERT-WWM, and ERNIE-Baidu.Source: §3.2 · p. 5
- Either ERNIE-Baidu 2.0 or NEZHA achieves the best performance in both BASE and LARGE model groups, though some comparisons may not be entirely fair.Some results were copied from original papers, and the authors note differences in experimental settings or finetuning methods; they also report consistent implementation gaps on CMRC.Source: Table 4 · p. 6, §3.2 · p. 6
3.3 Ablation Study
- Ablations compare positional encoding, whole word masking, training sequence length, and training-corpus sources, with the tested techniques generally contributing positively to downstream tasks.The variants use the NEZHA_BASE model.Source: §3.3 · p. 6
- Functional relative positional encoding shows a notable advantage, and with a 128-token training maximum, absolute encodings perform significantly worse than relative encodings on CMRC.The authors attribute the contrast to CMRC passages being much longer.Source: §3.3 · p. 7, Table 5 · p. 7
4 Conclusion
The report presents NEZHA, pretrained on large-scale Chinese corpora with functional relative positional encoding and other training techniques. The authors report state-of-the-art performance on several Chinese NLU tasks and plan to extend the work to other languages and scenarios.
4 Conclusion
- NEZHA combines functional relative positional encoding with whole word masking, mixed-precision training, and the LAMB optimizer for pretraining on Chinese corpora.The report says functional relative positional encoding leads to notable improvement over other positional encodings.Source: §4 · p. 7
- The authors report state-of-the-art performance on several Chinese natural language understanding tasks.The conclusion describes NEZHA’s performance across several tasks without specifying metrics here.Source: §4 · p. 7
- Future work plans to improve NEZHA for Chinese and other languages and extend its applications to more scenarios.Source: §4 · p. 7