Source-linked AI summary
MECT: Multi-Metadata Embedding based Cross-Transformer for Chinese Named Entity Recognition
Shuang Wu, Xiaoning Song, Zhenhua Feng
TL;DR
Chinese NER must handle word-boundary ambiguity, and Chinese character structure can provide semantic information. MECT fuses radical, character, and word information in a two-stream Cross-Transformer, and experiments on four benchmarks show that radical information can improve Chinese NER performance.
Problem
Chinese NER faces word-boundary ambiguity because Chinese text has no spaces between characters, while character structure can carry semantic information relevant to recognition.
Source: §1 · pp. 1–2Method
Results
Experiments on four benchmarks demonstrate that Chinese character radical information can effectively improve Chinese NER performance.
Source: §6 · p. 8Takeaways & Limitations
Radical information is a useful additional source of semantic information for Chinese NER.
Source: §6 · p. 8Adding the radical stream increases model complexity, motivating more efficient integration of character, word, and radical information.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Recently, word enhancement has become very popular for Chinese Named Entity Recognition (NER), reducing segmentation errors and increasing the semantic and boundary information of Chinese words. However, these methods tend to ignore the information of the Chinese character structure after integrating the lexical information. Chinese characters have evolved from pictographs since ancient times, and their structure often reflects more information about the characters. This paper presents a novel Multi-metadata Embedding based Cross-Transformer (MECT) to improve the performance of Chinese NER by fusing the structural information of Chinese characters. Specifically, we use multi-metadata embedding in a two-stream Transformer to integrate Chinese character features with the radical-level embedding. With the structural characteristics of Chinese characters, MECT can better capture the semantic information of Chinese characters for NER. The experimental results obtained on several well-known benchmarking datasets demonstrate the merits and superiority of the proposed MECT method.\footnote{The source code of the proposed method is publicly available at https://github.com/CoderMusou/MECT4CNER.
1 Introduction
Chinese NER faces word-boundary ambiguity, while Chinese character structure can contribute semantic information. MECT extends FLAT with a radical stream, and the paper reports improvements on benchmark datasets.
1 Introduction
- Chinese text has no spaces between characters, making word segmentation difficult for NER alongside challenges such as nested entities and indefinite entity length.The paper also identifies complex entity combinations as a challenge.Source: §1 · p. 1
- Chinese character structures, including radicals, head-and-tail forms, and structural components, can enrich character semantics for NER.The paper notes that Bi-LSTM-CRF used character-structure embeddings, but its sequential processing limits parallelism.Source: §1 · p. 2
- MECT extends FLAT with a radical stream to combine word-boundary and semantic learning with Chinese character structure information.The paper describes FLAT as offering efficient parallel computing and lexicon learning.Source: §1 · p. 2
- The proposed contributions include multi-metadata embedding and a two-stream model combining radical, character, and word information.The paper evaluates the method on several well-known Chinese NER benchmarks.Source: §1 · p. 2
- MECT is reported to demonstrate superiority over state-of-the-art methods on several well-known Chinese NER benchmarks.The introduction also reports improvement over the baseline on different benchmarks.Source: §1 · p. 2
2 Related Work
Prior Chinese NER enhancement work is presented in two broad strands: lexical information fusion and glyph-structural information fusion. The section describes representative approaches in both strands.
2 Related Work
- Lexical enhancement methods use word matching to add lexical information to character-based Chinese NER models.Examples include Lattice-LSTM, CNN, graph-network, Transformer-based, and label-and-probability approaches.Source: §2 · p. 2
- Glyph-structural methods incorporate character structure through radical-level embeddings or character-image features.Dong et al. (2016) used Bi-LSTM radical embeddings and reported state-of-the-art performance on MSRA; Glyce (Meng et al., 2019) extracted strokes and structure from character images.Source: §2 · p. 2
- Other studies also use radical information with Tencent’s pre-trained embedding to improve performance.The cited works are Xu et al. (2019) and Song et al. (2020).Source: §2 · p. 3
3 Background
FLAT adds word-lattice information to the Transformer encoder, using dictionary matching to represent word semantics and position boundaries. Its relative position encoding uses character head and tail positions.
3 Background
- FLAT augments a Transformer encoder with word-lattice information, including semantic and position-boundary information obtained through dictionary matching.Figure 1 presents FLAT’s input and output.Source: §3 · p. 3
- FLAT uses relative position encoding based on head and tail positions to represent word-boundary information.The encoding is calculated using a learnable parameter and concatenated head and tail positions.Source: §3 · p. 3
- The relative position encoding contributes to scaled dot-product attention, with position spans formed from head and tail position differences.The passage identifies u, v, and W2 as learnable parameters.Source: §3 · p. 3
4 The Proposed MECT Method
MECT extends FLAT with a two-stream network that integrates lattice information and Chinese character structure. A Cross-Transformer exchanges information between the streams, whose features are then fused for NER.
4 The Proposed MECT Method
- MECT uses a two-stream Transformer to integrate character-component information with FLAT’s word meaning and boundary information.A Cross-Transformer fuses the two metadata streams using a randomly initialized attention matrix as attention bias.Source: §4 · p. 3
4.1 CNN for Radical-level Embedding
- Because multiple structural components shape character meaning, MECT uses structural components (SC) and a CNN to produce radical-level character embeddings.Characters are decomposed into SC before CNN processing, followed by max-pooling and fully connected layers.Source: §4.1 · pp. 3–4
4.2 The Cross-Transformer Module
- Two Transformer encoders cross lattice and radical information, using contextual and lexical information to supplement character semantics.Linear transformations produce the lattice and radical Q/K/V inputs; FLAT relative-position encoding represents word boundaries in attention.Source: §4.2 · p. 4
4.3 Random Attention
- Randomly initialized attention improves MECT empirically, possibly because its bias adapts to scores in the lattice and radical subspaces.The random matrix is added to the previous attention scores.Source: §4.3 · pp. 4–5
4.4 The Fusion Method
- MECT concatenates lattice and radical features for fully connected fusion, then masks word positions and passes the fused features to a CRF.The fully connected layer’s parameters are learnable.Source: §4.4 · p. 5
5 Experimental Results
MECT is evaluated on four Chinese NER benchmarks and compared with existing methods; further analyses examine its attention, radical representations, efficiency, and component contributions.
5.1 Experimental Settings
- MECT was evaluated on Weibo, Resume, MSRA, and OntoNotes 4.0 using span-based F1, precision, and recall; SC was used as the radical feature and comes from online Xinhua.CNN used 30 one-dimensional convolution kernels of size 3, and SMAC was used to search for optimal hyperparameters.Source: §5 · p. 5, §5.1 · p. 5
5.2 Comparison with SOTA Methods
- 2.98% F1 improvement over the baseline on Weibo, with MECT reaching 61.91% on NE.The comparison reports F1 scores for NE, NM, and Overall.Source: Table 4 · p. 5, §5.2 · p. 6
- MECT exceeds other models and the baseline on Resume F1, gains 0.47% F1 over the baseline on OntoNotes 4.0, and outperforms existing SOTA methods overall on MSRA.The respective results are reported in Tables 5, 6, and 7.Source: §5.2 · p. 6, Table 5 · p. 6, Table 6 · p. 6
- MECT further improves BERT performance significantly across the four datasets.The BERT results occupy the fourth block of each comparison table.Source: §5.2 · p. 6
5.3 Effectiveness of Cross-Transformer
- Lattice attention emphasizes word-character relations and boundary information, while radical attention focuses globally and corrects character semantics; the authors describe them as complementary.Figure 4 visualizes the two attention-score maps.Source: §5.3 · pp. 6–7, Figure 4 · p. 7
5.4 Impact of Radicals
- Characters sharing a radical or similar structure have smaller cosine distances in the learned embeddings; MECT also avoids some FLAT labeling errors, and PER-label accuracy rises from 81.6% to 86.96%.The reported examples include percentages mislabeled as PER and lexicon entries mistakenly treated as valid words by FLAT.Source: §5.4 · p. 7, Figure 5 · p. 7
5.5 Analysis in Efficiency and Model Size
- On an NVIDIA GeForce RTX 2080Ti, MECT's parallel inference speed is reduced by 0.15 relative to the stated comparison, and its model has 2–4 million parameters.Inference tests use batch sizes 16 and 1; parameter count depends on maximum sentence length and d_model size.Source: §5.5 · p. 7, Figure 6 · p. 8
5.6 Ablation Study
- Ablations report better performance for the two-stream model, stronger feature extraction with exchanged query vectors, and drops across datasets when random attention is removed.The authors describe random attention as an attention bias that can eliminate differences caused by different embeddings.Source: §5.6 · p. 8, Table 8 · p. 8
6 Conclusion
MECT is a two-stream network that fuses radical, character, and word information through multi-metadata embedding and a Cross-Transformer. Experiments on four benchmarks support the effectiveness of radical information for Chinese NER, while the radical stream increases model complexity.
6 Conclusion
- Experiments on four benchmarks demonstrate that radical information can effectively improve Chinese NER performance.MECT fuses radicals, characters, and words through multi-metadata embedding in a two-stream Cross-Transformer network.Source: §6 · p. 8
- Adding the radical stream increases MECT’s model complexity.The authors identify more efficient integration of character, word, and radical information as future work.Source: §6 · p. 8
- The authors plan to explore extending more efficient two- or multi-stream integration to other NLP tasks.Source: §6 · p. 8
A.1 Range of Hyper-parameters
Hyperparameter selection differed by dataset size: parameters were manually selected for Ontonotes 4.0 and MSRA, while SMAC searched for the best hyperparameters for Weibo and Resume. The search ranges are listed in Table 9.
A.1 Range of Hyper-parameters
- Parameters were manually selected for the two large datasets, Ontonotes 4.0 and MSRA.Source: §A.1 · p. 11
- SMAC was used to search for the best hyperparameters on the two small datasets, Weibo and Resume.Source: §A.1 · p. 11
- Table 9 lists the hyperparameter search ranges.Source: Table 9 · p. 11, §A.1 · p. 11