Source-linked AI summary
Simplify the Usage of Lexicon in Chinese NER
Ruotian Ma, Minlong Peng, Qi Zhang, Xuanjing Huang
TL;DR
Chinese NER benefits from word lexicons, but Lattice-LSTM’s complex architecture limits efficiency and transfer to other neural models. The paper introduces SoftLexicon, which adds lexicon information through character representations, and reports faster inference with better performance across four benchmark datasets.
Problem
Chinese NER needs word information because character-only models do not fully exploit lexicons, while Lattice-LSTM is complex and inefficient for real-time use.
Method
The paper encodes all lexicon-matched words into character representations, requiring only a character-representation adjustment that can transfer across neural NER architectures.
Results
Across four benchmark Chinese NER datasets, the method achieves faster inference and better performance than compared state-of-the-art methods.
Takeaways & Limitations
The method provides a simpler way to incorporate lexicons and can be easily combined with pre-trained models such as BERT.
Takeaways & Limitations
ExSoftword loses matching information and cannot introduce pre-trained word embeddings, motivating a frequency-based weighting algorithm for better lexicon use.
Abstract
from arXiv · showhide
Recently, many works have tried to augment the performance of Chinese named entity recognition (NER) using word lexicons. As a representative, Lattice-LSTM (Zhang and Yang, 2018) has achieved new benchmark results on several public Chinese NER datasets. However, Lattice-LSTM has a complex model architecture. This limits its application in many industrial areas where real-time NER responses are needed. In this work, we propose a simple but effective method for incorporating the word lexicon into the character representations. This method avoids designing a complicated sequence modeling architecture, and for any neural NER model, it requires only subtle adjustment of the character representation layer to introduce the lexicon information. Experimental studies on four benchmark Chinese NER datasets show that our method achieves an inference speed up to 6.15 times faster than those of state-ofthe-art methods, along with a better performance. The experimental results also show that the proposed method can be easily incorporated with pre-trained models like BERT.
1 Introduction
Chinese NER is difficult because Chinese sentences are not naturally segmented, while lexicon-enhanced models can be accurate but operationally complex. The proposed method incorporates lexicon information into character representations, improving speed, transferability, and performance.
- Motivation: Chinese NER is harder than English NER because Chinese sentences are not naturally segmented.Word segmentation errors can affect entity boundaries and category prediction.
- Limitations of prior work: Lattice-LSTM preserves multiple lexicon matches but requires a complicated sequence-modeling architecture.It adds edges between nonadjacent characters to incorporate lexicon information.
- Limitations of prior work: Lattice-LSTM’s complexity slows training and inference and makes transfer to CNNs and transformers difficult.These constraints limit use in applications requiring real-time NER responses.
- Proposed method: The proposed method encodes all matched lexicon words in character representations instead of modifying the sequence-modeling architecture.It requires only an adjustment to the character representation layer and is designed to preserve lexicon matching results.
- Proposed method: The method is transferable across neural sequence-labeling architectures and can be incorporated with pre-trained models such as BERT.The contribution specifically emphasizes adaptation through the character representation layer.
- Results: Across four public Chinese NER datasets, a single-layer Bi-LSTM implementation improves inference speed and sequence-labeling performance over state-of-the-art methods.The supplied result passage reports improvements in both dimensions without specifying their numerical values.
2 Background
Prior lexicon and segmentation approaches preserve useful word information but introduce either segmentation errors or substantial architectural and computational complexity. Lattice-LSTM retains multiple matches and word embeddings, yet its graph-based implementation is difficult to parallelize and limits broader deployment.
- Softword Feature: Softword augments each character representation with the embedding of its predicted segmentation label.Typical segmentation labels are B, M, E, and S.
- Softword Feature: Softword can inevitably introduce segmentation errors because gold segmentation is unavailable in most datasets and predicted segmentation may be incorrect.These errors enter the downstream approach through the segmenter’s output.
- Lattice-LSTM: Lattice-LSTM performs lexicon matching and preserves all matches by connecting characters with multiple directed edges, converting the sentence chain into a graph.Each matching subsequence ending at a character contributes corresponding hidden states and memory cells.
- Lattice-LSTM: Lattice-LSTM modifies the normal LSTM to update character states using lists associated with matching subsequences.The function f is described as a simplified representation of its memory-update function.
- Lattice-LSTM: Lattice-LSTM preserves possible matches and introduces pre-trained word embeddings, which are identified as advantages over heuristic single-match selection.These design choices retain more lexicon information for the NER model.
- Lattice-LSTM: Lattice-LSTM’s additional state lists and complicated memory-update function slow training and inference, while its implementation hinders parallel processing.The published implementation used batch size 1, and these issues limit real-time industrial applications.
3 Approach
SoftLexicon incorporates lexicon information by adjusting character representations rather than complicating sequence modeling. It preserves matched-word information through BMES categorization, weighted condensation, concatenation, and standard sequence modeling.
- Character Representation Layer: SoftLexicon introduces lexicon information by adjusting each character representation before sequence modeling and CRF prediction.The architecture maps characters to dense vectors, adds SoftLexicon features, then applies sequence modeling and CRF inference.
- Incorporating Lexicon Information: ExSoftword retains possible segmentation labels with a multi-hot vector, but cannot restore the original matching results uniquely.Characters may have multiple matching-word configurations corresponding to the same constructed feature sequence.
- Incorporating Lexicon Information: The method categorizes every matched lexicon word for a character into four BMES word sets, adding NONE when a set is empty.This organization retains segmentation information while representing multiple possible matches.
- Incorporating Lexicon Information: SoftLexicon condenses each BMES word set into a fixed-dimensional vector using mean pooling or frequency-based weighting.The weighting approach uses offline word frequencies rather than dynamic attention to maintain computational efficiency.
- Sequence Modeling Layer: The four condensed word-set representations are concatenated and added to each character representation.This fixed-dimensional feature is then processed by a sequence-modeling layer, implemented here with a single-layer BiLSTM, followed by CRF inference.
4 Experiments
Experiments on four Chinese NER datasets evaluate SoftLexicon’s speed, performance, transferability across sequence-modeling architectures, and compatibility with BERT. The method is consistently more efficient than key baselines and improves performance across the reported settings.
- Experiment Setup: The experiments follow Lattice-LSTM protocols, using four datasets, standard baselines and P/R/F1 evaluation metrics.The datasets are OntoNotes, MSRA, Weibo NER and Resume NER.
- Computational Efficiency Study: 6.15 times faster than Lattice-LSTM, SoftLexicon achieves higher inference efficiency with the same batch size of 1.The comparison includes Lattice-LSTM, LR-CNN and BERT-based systems, evaluated by average sentences processed per second on a GPU.
- Computational Efficiency Study: SoftLexicon remains faster than Lattice-LSTM and LR-CNN as sentence length increases, despite speed degradation from the recurrent LSTM architecture.The speed advantage is especially pronounced for short sentences and remains consistent across longer inputs.
- Effectiveness Study: On OntoNotes, SoftLexicon outperforms Lattice-LSTM by 1.76% F1 and performs comparably with word-based methods using gold segmentation.Replacing gold segmentation with automatic segmentation lowers F1 from 75.77% to 71.70%, while SoftLexicon avoids relying on predicted segmentation.
- Transferability Study: Across different neural sequence-modeling layers, SoftLexicon consistently outperforms corresponding ExSoftword baselines.The LSTM-based architecture performs better than the CNN- and transformer-based variants in the reported comparison.
- Combining Pre-trained Model: SoftLexicon with BERT outperforms the BERT tagger on all four datasets, showing that lexicon information complements contextual representations.The BERT encoder outputs are concatenated with the character representations before sequence labeling.
- Ablation Study: Removing the Middle word group, BMES distinctions, or overall weight normalization degrades performance across the ablation studies.These results support retaining internal matches, distinguishing matched-word categories, and normalizing weights across matched words.
5 Conclusion
The paper addresses the computational efficiency of incorporating word lexicons into Chinese NER. Across four benchmark datasets, the proposed character-representation method achieves faster inference and better performance than the compared state-of-the-art methods.
- Conclusion: Across four benchmark Chinese NER datasets, the proposed method achieves faster inference and better performance than the compared state-of-the-art methods.The method incorporates lexicon information into character representations to address computational efficiency.