Source-linked AI summary
Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter
Wei Liu, Xiyan Fu, Yue Zhang, Wenming Xiao
TL;DR
Chinese sequence labeling lacks explicit word boundaries, and existing lexicon–BERT methods do not integrate external features into BERT’s lower layers. LEBERT inserts a Lexicon Adapter between BERT Transformer layers and achieves state-of-the-art performance across ten datasets covering NER, Word Segmentation, and POS tagging.
Problem
Existing lexicon–BERT methods fuse external lexicon features in a shallow sequence layer rather than integrating them into BERT’s bottom layers, limiting their use of BERT representations.
Method
LEBERT converts sentences into character-word pairs and inserts a Lexicon Adapter between BERT Transformer layers to integrate lexicon knowledge directly.
Results
LEBERT achieves state-of-the-art performance on ten datasets spanning Chinese NER, Word Segmentation, and POS tagging.
Takeaways & Limitations
Bottom-level fusion of lexicon features and BERT representations is effective across the three evaluated Chinese sequence labeling tasks.
Takeaways & Limitations
The approach assumes that directly injecting lexicon features into BERT can affect performance because lexicon and BERT representations differ, motivating BERT fine-tuning.
Abstract
from arXiv · showhide
Lexicon information and pre-trained models, such as BERT, have been combined to explore Chinese sequence labelling tasks due to their respective strengths. However, existing methods solely fuse lexicon features via a shallow and random initialized sequence layer and do not integrate them into the bottom layers of BERT. In this paper, we propose Lexicon Enhanced BERT (LEBERT) for Chinese sequence labelling, which integrates external lexicon knowledge into BERT layers directly by a Lexicon Adapter layer. Compared with the existing methods, our model facilitates deep lexicon knowledge fusion at the lower layers of BERT. Experiments on ten Chinese datasets of three tasks including Named Entity Recognition, Word Segmentation, and Part-of-Speech tagging, show that LEBERT achieves the state-of-the-art results.
1 Introduction
Chinese sequence labeling is difficult because sentences lack explicit word boundaries, while existing lexicon–BERT methods fuse external features only at a shallow model level. LEBERT instead integrates lexicon information between BERT Transformer layers and achieves state-of-the-art results across three tasks and ten datasets.
- Chinese sequence labeling is challenging because sentences lack explicit word boundaries, and segmentation-first pipelines can propagate CWS errors.
- Existing lexicon–BERT approaches combine contextual BERT representations and lexicon features in a shallow sequence-labeling layer rather than BERT’s bottom layers.
- LEBERT converts sentences into character-word pairs and uses a lexicon adapter with char-to-word bilinear attention between adjacent BERT Transformer layers.
- LEBERT fine-tunes both BERT and the lexicon adapter so word information can interact with BERT representations throughout the multilayer encoder.
- State-of-the-art performance is achieved on all ten benchmark datasets spanning Chinese NER, Word Segmentation, and POS tagging.
2 Related Work
Prior work enhances character-based Chinese sequence labeling with lexicons, pretrained models, shallow fusion, or lexicon-guided pretraining. LEBERT differs by inserting an adapter between BERT layers to fuse lexicon information directly, without requiring raw texts or entity sets.
- Lexicon-based methods enhance character models by encoding matched words alongside characters for Chinese NER, CWS, and POS tagging.
- BERT-based methods use Transformer-derived character features and have achieved strong results for Chinese sequence labeling tasks.
- Hybrid methods commonly concatenate BERT representations and lexicon information in a shallow or model-level fusion layer.
- Lexicon-guided pretraining methods include ERNIE’s implicit masking approach and ZEN’s multilayer N-gram encoder, whose vocabulary size is limited.
- LEBERT requires no raw texts or entity set for lexicon integration, unlike the described lexicon-guided pretraining approaches.
- LEBERT integrates lexicon information with an adapter between BERT Transformer layers, targeting bottom-level fusion rather than parameter-efficient training.
3 Method
LEBERT converts Chinese sentences into character–word pair sequences and injects lexicon information directly between BERT Transformer layers through a Lexicon Adapter. The adapter attends over matched words, adds weighted lexicon information to character representations, and feeds the result through BERT and a CRF decoder.
- 3 Method: LEBERT differs from BERT by taking character and lexicon features as input and attaching a Lexicon Adapter between Transformer layers.The adapter enables lexicon knowledge to be integrated into BERT rather than fused only at the final sequence-labeling layer.
- 3.1 Char-Words Pair Sequence: A character-words pair sequence assigns every matched lexicon word to the characters it contains.Potential words are retrieved by matching sentence subsequences against a Trie, then represented as pairs (c_i, ws_i).
- 3.2 Lexicon Adapter: The Lexicon Adapter uses a character vector and paired word embeddings to compute word relevance with character-to-word bilinear attention.Word embeddings are transformed to align with BERT’s hidden size before attention-based weighting.
- 3.2 Lexicon Adapter: The adapter adds the weighted lexicon vector to the character-level vector, then applies dropout and layer normalization.This operation injects the selected lexicon information into the character representation.
- 3.3 Lexicon Enhanced BERT: After k Transformer layers, LEBERT applies the adapter to each character–word pair and passes the transformed representations through the remaining BERT layers.The paper specifies L = 12 Transformer layers and uses the final hidden output for sequence labeling.
- 3.4 Training and Decoding: A CRF layer models dependencies between successive labels, trains with sentence-level negative log-likelihood, and decodes using the Viterbi algorithm.The CRF assigns probabilities to complete label sequences using transition scores.
4 Experiments
LEBERT is evaluated against BERT, lexicon-enhanced, hybrid, and state-of-the-art systems across ten datasets covering Chinese NER, word segmentation, and POS tagging. Results support BERT-level lexicon fusion and analyze span behavior, sentence length, adapter placement, and fine-tuning.
- Experimental Settings: Ten datasets cover Chinese NER, Chinese Word Segmentation, and Chinese POS tagging, with standard F1-score as the evaluation metric.
- Overall Results: LEBERT improves F1 on all four Chinese NER datasets over model-level fusion systems, including BERT+Word and other shallow-fusion baselines.
- Overall Results: LEBERT achieves the best results for Chinese Word Segmentation and outperforms both BERT+Word and lexicon-guided pre-training methods.
- Overall Results: LEBERT achieves state-of-the-art results across all datasets, with significant relative error reductions over BERT and BERT-based state-of-the-art models.
- Model-level Fusion vs. BERT-level Fusion: Both BERT+Word and LEBERT outperform BERT on Span F1 and Type Acc, while LEBERT gains more than BERT+Word on OntoNotes and UD1.
- Model-level Fusion vs. BERT-level Fusion: On OntoNotes, F1 declines as sentence length increases for all models, while lower-layer fusion is associated with capturing more complex semantics and selecting relevant words.
- Discussion: Without fine-tuning BERT, F1 decreases by 7.03 points on OntoNotes and 3.75 points on UD1, indicating fine-tuning is important for lexicon integration.
5 Conclusion
LEBERT integrates lexicon features directly between BERT Transformer layers through a Lexicon Adapter, enabling in-depth fusion at the BERT level. Experiments show state-of-the-art performance across ten datasets spanning three Chinese sequence labeling tasks.
- LEBERT injects lexicon information between BERT Transformer layers using a Lexicon Adapter.
- LEBERT enables in-depth fusion of lexicon features with BERT representations at the BERT level.
- LEBERT achieves state-of-the-art performance on ten datasets covering three Chinese sequence labeling tasks.